Reliability

Is 'reliability' the right word to use for qualitative research? Cohen et al. (2007, Ch.10) suggest a variety of different words that should be used to distinguish the variable-controlled nature of quantitative research from the variable-embracing nature of qualitative research. Points addressing reliability that are present in my research include repeating the method of data collection and analysis across several groups (related to triangulation, below) and using anonymous questionnaires to encourage honesty amongst participants (p.209).

Early on in my research plan I discussed some of the principles laid out by Newby (2010, pp.120-3) for ensuring some degree of reliability in qualitative research, particularly the concept of triangulation. In the case of my project, the point about getting information form multiple sources is the best source of reliability. The questionnaire has been answered by participants from different cohorts of the same program, so we can gain some idea of how reliable (or not) the results are by comparing different cohort responses. Although the research is qualitative in nature, the reliability can be commented on by comparing the number of responses with the actual number of participants for each cohort.

References

• Cohen, L., Manion, L., Morrison, K. & Bell, R. (2007). Research methods in education. Routledge Limited.
• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.

Validity

There is a danger of intertwining the concepts of reliability and validity, so I'm attempting to address them in separate posts.  Cohen, Manion & Morrison (2013, pp. 177-99) devote a great deal of attention to the concept of validity, drawing on a variety of sources to lend weight and richness to the discussion.  Several of the points (p.180) raised are applicable to my research data:

• The natural setting is the principle source of data
• Context-boundedness and 'thick description'
• Data are socially situated
• The researcher is part of the researched world
• The researcher - rather than the research tool - is the key instrument of research
• The data are descriptive
• There is a concern for processes rather than simply with outcomes (inherent in my choice of action research)
• Data are analysed inductively rather than using a priori categories
• Data are presented in terms of the respondents rather than researchers

There are some points to beware of that might undermine the validity of my research - I will also list how these can be addressed:

• Reactivity (internal validity) - I am aiming to improve the processes involved in the learning programs; this should not affect the data for previous cohorts, although the current cohort could potentially be affected by knowing that they are being observed.  This also touches on the issue of Researcher bias mentioned by the authors
• Concensual validity (external) - since I am undertaking this research as a learning experience, will 'competent others' dismiss my findings due to my inexperience, or because they think I'm simply forcing the data to fit so that I pass my assessment?

Triangulation is dealt with as a means of ensuring validity.  Newby (2010, pp. 121-3) also mentions this technique, although in somewhat less detail.  So to what extent do my research methods yield themselves to this?

• Time triangulation - I have used the same method for a number of cohorts, although this effect may be diminished because I did not carry out the survey for each group immediately following their participation, so earlier groups may not recall their experiences as accurately.
• Theoretical triangulation - since I have avoided basing my questions on one particular theory, there is the opportunity to compare the results from the point of view of competing theories for social and online learning.
• Investigator triangulation - the data are recorded electronically, so potentially other researchers could give their own interpretations.
• Methodological triangulation - the same method has been used on different groups, so I can easily compare the results of each group to consider how well the results support conclusions for each group.

General points that have contributed to validity on this project:

• Choosing an appropriate methodology for answering the research question - action research.  This has ensured that the focus is on processes - not outcomes, which I might be interested in unfairly interpreting!
• Selecting appropriate instrumentation - using online questionnaires allowed the data to be gathered according to the time needs of respondents, and allowing them access to reminders (their forum postings) whilst responding to questions.

Limitations here include:

• Sampling - by not requiring participation in the survey, I sacrificed control over sample sizes, which could potentially limit the validity (and reliability) of results.  However I considered that sensitivity to participants' wishes was of greater importance in this case, since some people are uncomfortable with the use of online forums in the first place.

Overall I believe I have sufficiently addressed issues of validity, but there are clearly many others that I have not encountered yet, including the points concerning data interpretation.

References

• Cohen, L., Manion, L. & Morrison, K. (2007) Research Methods in Education.
• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.

Blog Post 2: Quantitative vs Qualitative

Originally posted to eBridge, 11 February 2013

Method of searching for ‘truth’

Quantitative research demands that there be a single objective truth that can be determined from the research – for deducing empirical values (constants of the universe, material properties) this is without doubt a valid method! However for anything that involves people we risk skewing our research by testing for a positive hypothesis – we need to consider that multiple factors will be in play, therefore we should start by testing a null hypothesis (Newby, 2010, p.99; Cohen & Manion, Ch.1).

Quantitative research has its basis in deducing the truth from a set of pre-conceived logic – determined largely by our own experience of the world. This can only work for as long as our intuition holds. We will eventually hit an upper limit – and then we need to turn to inductive research to push the boundaries forward. Here we rely on gathering data and spotting patterns that demand explanation. For real research, we actually use a mixture of methods, formulating deductive hypotheses and testing these against inductive reasoning based on data.

Researcher standpoint

Quantitative research seems to demand that the researcher should be neutral in their standpoint, since they are collecting data to prove or disprove a hypothesis, and cannot take actions that might skew the results. The researcher has to passively observe subjects in a ‘laboratory style’ setting. By contrast, qualitative researchers seek a natural setting to ensure that all variables are free to act on the system. The researcher may have some commitment, especially if some kind of change to existing practice is being urged.

Data – format & collection method

Quantitative analysis is primarily concerned with the collection of numerical data, to give confirmation of hypotheses. There will be methods for collection that are determined before the study commences – these will be formalised, especially if the study is to be carried out on a large scale, to prove that the hypothesis is universally true. Qualitative research allows for any kind of data to be included in the study, bringing context into the matter. As the researcher is not expected to be completely neutral, the formality is relaxed, although some structure must be observed to ensure that the results will be respected by a community.

Learning points

Taking a good critical look at the philosophy behind research has helped to identify how my science background has railroaded me into the quantitative mind-set, but without ever having had any real understanding of the assumptions imposed. Furthermore I wasn't involved in setting out a research proposal in the first instance, since projects were pre-approved for funding. Without having enough professional experience, or any connections beyond the university, there was no way to consolidate my learning into any kind of useful skill that could be transferred outside of the institution.

By considering the different factors that are working beneath the surface as I consider my research project, I will be able to transform my own practice in learning & development, as well as learning some transferable research skills at the same time. This makes the project potentially both action and evaluation research at the same time.

References:

• Cohen, L., Manion, L. & Morrison, K. (2007). Research Methods in Education
• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.