Selection of research method

Draft section of my reflective assignment, based on the blog post ‘A little more action (research) please!’

I have been forced to challenge my own pre-disposition towards quantitative research methods, which was influenced by my physical sciences background. I identified this bias at an early stage in my writing, and found the distinction with qualitative research better defined in my mind by reading the comparisons made by Creswell (2009, Ch.7) and Newby (2010, Ch.3). I also realised that my previous experiences lacked any real involvement in the formal planning of research; my previous projects has always been funded without me having to submit research proposals myself. I decided that qualitative approaches seemed better suited to my context, but it took some time to fully challenge my unconscious habits. I was able to identify possible sources of bias towards theoretical models that I had used (Salmon, 2004).

However it took some additional reading (and re-reading) to fully isolate my unconscious assumptions. After extensive reading about how to create both quantitative and qualitative research proposals, I believed that I had created a set of reasonable questions for qualitative research. My initial research questions were phrased as ‘What effect does...’ and ‘How does...affect...’ Only by revisiting some of the initial reading did I notice that, despite my initial conclusions that qualitative research would be the best approach, that I had automatically designed my questions in a directive way, that would lead to bias towards theory rather than interpreting participants responses from a neutral standpoint.

After some more reading, I was able to present a much more complete and reasoned overview of my research proposal, showing a great deal more thought. This revised research plan fitted much more closely with the principles of action research (Creswell, 2009; Newby, 2010 pp.623-4; O’Brien, 1998), and represents a successful change in my thinking about research.

‘Re-learning means to abolish some toxic assumptions’ (Leonhard, 2013)

References:

• Creswell, J. W. (2009). Research Design: qualitative, quantitative and mixed methods approaches. (3rd edition) Sage.
• Leonhard, G. (2013). Beyond the obvious: re-defining the meaning of learning in a networked society. (video online) Available at: https://www.annotag.tv/learningtechnologies/play/18320
• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.
• O'Brien, R. (1998). An Overview of the Methodological Approach of Action Research. (online) Available at: http://www.web.ca/~robrien/papers/xx%20ar%20final.htm (Accessed March 2013)
• Salmon, G. (2004). e-Moderating: The Key to Teaching and Learning Online. 2nd ed. London: Routledge-Falmer.

Reliability

Is 'reliability' the right word to use for qualitative research? Cohen et al. (2007, Ch.10) suggest a variety of different words that should be used to distinguish the variable-controlled nature of quantitative research from the variable-embracing nature of qualitative research. Points addressing reliability that are present in my research include repeating the method of data collection and analysis across several groups (related to triangulation, below) and using anonymous questionnaires to encourage honesty amongst participants (p.209).

Early on in my research plan I discussed some of the principles laid out by Newby (2010, pp.120-3) for ensuring some degree of reliability in qualitative research, particularly the concept of triangulation. In the case of my project, the point about getting information form multiple sources is the best source of reliability. The questionnaire has been answered by participants from different cohorts of the same program, so we can gain some idea of how reliable (or not) the results are by comparing different cohort responses. Although the research is qualitative in nature, the reliability can be commented on by comparing the number of responses with the actual number of participants for each cohort.

References

• Cohen, L., Manion, L., Morrison, K. & Bell, R. (2007). Research methods in education. Routledge Limited.
• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.

Data analysis

Although I set out to do a qualitative research project, I did reserve the right to quantify my results to some extent. I've found it helpful when making sense of my data to do a quick finger-test of whether comments were positive, neutral or negative, and summarise them by question and group. These can then be totalled and presented graphically, which will be very helpful when sharing back with participants. The idea of sharing with participants as a first audience seems particularly prudent to avoid accusations of 'intellectual imperialism' (Newby, 2010, p.48). This will also act as an important feedback exercise to help practitioners see the reaction of participants, and to give me more direct contact with both parties to better understand how to inductively expand the research.

There is still a lot more work to do on full data analysis at this stage, but deadlines are looming!

References:

• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.

Collecting the data

I've been a bit lax in following the last stages of the pathway, so I thought I had better get some thoughts down before they fade away. 

What challenges do you face and what successes do you have?

Challenges:

• Some of the experiences I am asking people to recount are in some cases from over 2 years ago.
• The return rate for the questionnaire has been around 50% of all possible participants - partly this is due to it being a busy time of year for staff.

Successes:

• I have received a good number of responses, with an excellent level of detail
• Responses have yielded insights into how to improve the processes in question

What unexpected experiences do you have?

The use of an online questionnaire has left me feeling somewhat disembodied - most of the methods described in our reading concerned face-to-face methods. However they have been very effective, and in keeping with the idea that online learning should be anytime, anywhere...

How does your experience of data collection reflect, or conflict with, the principles and theories encountered in your reading?

I realise from reading through Newby (2010, pp.332-3) that I didn't pilot the questionnaire as rigorously as I should have done.  My testing was limited to sending the link to one participant who I asked about trialling it beforehand. Fortunately I had made significant revisions to my questions early on in the project that kept the research on track.

References:

• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.

The nature of academic publishing

Image: forwardcom via freeimages

Building up towards my reflective assignment, I thought it might help to draft this in sections, according to particular areas of interest throughout the module. First stop is publishing...

One of the assumptions that I am interested in challenging is the area of academic publishing. We tend to regard the world of academic journals as ‘an open and public forum’ (Colquhoon, 2006) that is created for the common good, but a growing number of people are beginning to question if this holds true, especially in the age of open web content. Wheeler (2011a, b, c) has written a number of blog posts criticising the traditional system of academic publishing, noting that:

‘It’s a hierarchy that rarely changes.’

Furthermore, the hierarchical nature of the publishing world is often self-defeating in terms of securing the widest possible audience:

‘...many of the top, elite journals we are expected to publish in are in fact read by a very small percentage of the community the research is intended to reach.’

From considering my own worldview, I have noted the restrictions that hierarchical institutions place on the education system, particularly when combined with the solutions offered by market organisations (Hobson, 2012; Ronfeldt, 2012b). Potential disruptive alternatives may yet be found in the emerging network sector predicted by Ronfeldt (2012a), or the peer-to-peer ideology (Bauwens, 2009), although sustainable business models are yet to emerge. It should be noted that there are dangers inherent in the misuse of data for either solution. Newby (2010) warns of ‘intellectual imperialism’ when researchers are able to claim sole ownership of research data, discounting the rights of those involved in the study. At the other extreme of open web publishing, Lanier (2006a, b; 2012) warns of the dangers of ‘digital Maoism’, whereby all ownership and authorship of published data is lost, and transferred to control of the crowd.

References:

• Bauwens, M. (2009). Open Process academic publishing: towards the peer production of academic peer review. P2P Foundation's blog [blog] 15 September. Available at: <http://blog.p2pfoundation.net/open-process-academic-publishing-towards-the-peer-production-of-academic-peer-review/2009/09/15>
• Colquhoon, D. (2006). Research Methods in Education Contexts. University of Hull
• Hobson, J. (2012). The problems with educating a quadriform society. Thinking and learning in the digital age [blog] 7 August. Available at: <http://jimmyhob.blogspot.co.uk/2012/08/the-problems-with-educating-quadriform.html>
• Lanier, J. (2006a). Digital Maoism: The hazards of the new online collectivism. Edge.org [online] 30 May. Available at: <http://www.edge.org/3rd_culture/lanier06/lanier06_index.html>
• Lanier, J. (2006b). Beware the online collective. Edge.org [online] 25 December. Available at: <http://www.edge.org/3rd_culture/lanier06/lanier06.2_index.html>
• Lanier, J. (2012). Are we at the beginning of the rise of post human machine intelligence? [Video online] Available at: <http://dpcloud.co/v3/enterprise/learningtech/index.php?option=com_dpcloud&v=play&p=7117&u=learningtech&cat=Keynotes&Itemid=2>
• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.
• Ronfeldt, D. (2012a). TIMN in 20 minutes: a video presentation. Visions from two theories [blog] 23 May. Available at: <http://twotheories.blogspot.co.uk/2012_05_01_archive.html>
• Ronfledt, D. (2012b). Q’s & A’s about “TIMN in 20 minutes” (4th of 7): creation of a new sector? Visions from two theories [blog] 29 August. Available at: <http://twotheories.blogspot.co.uk/2012/10/qs-as-about-timn-in-20-minutes-4th-of-7_29.html>
• Wheeler, S. (2011a). Sharp practice. Learning with e's [blog] 26 September. Available at: <http://steve-wheeler.blogspot.co.uk/2011/09/sharp-practice.html>
• Wheeler, S. (2011b). The open case. Learning with e's [blog] 2 October. Available at: <http://steve-wheeler.blogspot.co.uk/2011/10/open-case.html>
• Wheeler, S. (2011c). When the dam breaks… Learning with e's [blog] 18 October. Available at: <http://steve-wheeler.blogspot.co.uk/2011/10/when-dam-breaks.html>

Validity

There is a danger of intertwining the concepts of reliability and validity, so I'm attempting to address them in separate posts.  Cohen, Manion & Morrison (2013, pp. 177-99) devote a great deal of attention to the concept of validity, drawing on a variety of sources to lend weight and richness to the discussion.  Several of the points (p.180) raised are applicable to my research data:

• The natural setting is the principle source of data
• Context-boundedness and 'thick description'
• Data are socially situated
• The researcher is part of the researched world
• The researcher - rather than the research tool - is the key instrument of research
• The data are descriptive
• There is a concern for processes rather than simply with outcomes (inherent in my choice of action research)
• Data are analysed inductively rather than using a priori categories
• Data are presented in terms of the respondents rather than researchers

There are some points to beware of that might undermine the validity of my research - I will also list how these can be addressed:

• Reactivity (internal validity) - I am aiming to improve the processes involved in the learning programs; this should not affect the data for previous cohorts, although the current cohort could potentially be affected by knowing that they are being observed.  This also touches on the issue of Researcher bias mentioned by the authors
• Concensual validity (external) - since I am undertaking this research as a learning experience, will 'competent others' dismiss my findings due to my inexperience, or because they think I'm simply forcing the data to fit so that I pass my assessment?

Triangulation is dealt with as a means of ensuring validity.  Newby (2010, pp. 121-3) also mentions this technique, although in somewhat less detail.  So to what extent do my research methods yield themselves to this?

• Time triangulation - I have used the same method for a number of cohorts, although this effect may be diminished because I did not carry out the survey for each group immediately following their participation, so earlier groups may not recall their experiences as accurately.
• Theoretical triangulation - since I have avoided basing my questions on one particular theory, there is the opportunity to compare the results from the point of view of competing theories for social and online learning.
• Investigator triangulation - the data are recorded electronically, so potentially other researchers could give their own interpretations.
• Methodological triangulation - the same method has been used on different groups, so I can easily compare the results of each group to consider how well the results support conclusions for each group.

General points that have contributed to validity on this project:

• Choosing an appropriate methodology for answering the research question - action research.  This has ensured that the focus is on processes - not outcomes, which I might be interested in unfairly interpreting!
• Selecting appropriate instrumentation - using online questionnaires allowed the data to be gathered according to the time needs of respondents, and allowing them access to reminders (their forum postings) whilst responding to questions.

Limitations here include:

• Sampling - by not requiring participation in the survey, I sacrificed control over sample sizes, which could potentially limit the validity (and reliability) of results.  However I considered that sensitivity to participants' wishes was of greater importance in this case, since some people are uncomfortable with the use of online forums in the first place.

Overall I believe I have sufficiently addressed issues of validity, but there are clearly many others that I have not encountered yet, including the points concerning data interpretation.

References

• Cohen, L., Manion, L. & Morrison, K. (2007) Research Methods in Education.
• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.

Revisiting ethics

Some more thoughts on the ethics of my project, based on the principles outline by Newby (2010, pp.47-9)

Moral position - None of the groups involved in the study will be treated differently, since the actual participation in learning programs has already taken place. However the responses will help to guide myself and my colleagues to ensure that the use of forums is not a detriment to future learning experiences, and ensure that problems experienced by individuals can be addressed.

Honesty - I am not aiming to fit the data to a particular theory or framework from the outset, although I will have to be careful to avoid the temptation to do this. In order to provide real justification for this, such results would have to be reliable and reproducible across several data sets, which is not likely to be possible across such small samples. By choosing action research I am committing to developing understanding of the online environment for practitioners & future participants alike.

Responsibility - All information shared is within the same organisation, and responses are anonymous, with the option to avoid being quoted in written reports, so nobody should be disadvantaged as a result of this study. There are no questions that I would expect to reveal anything sensitive, however this is something to bear in mind for future studies.

Sharing - I have expressed my intent to share results with participants, and with the wider organisation. What I will need to consider is the appropriate format to write up my results for others to appreciate. I'm sure most colleagues won't get much benefit from an academic-style paper! Nor should I be disseminating detailed results outside the organisation, although there is probably some benefit in talking about the overall process on a wider platform - this blog to be precise!

Freedom - Participants have had freedom to answer the questionnaire or not, and have also been given the option to have their responses omitted from written reports. I have not based the study on their actual forum entries, and I would be inclined to avoid doing so at such an early stage in the use of forums. People are apprehensive enough about engaging with the medium without the idea of their every word being used in evidence!

Community - This research project has been discussed with colleagues in Learning & Development to ensure that the nature of questions asked is appropriate, and that participants can benefit from participating.

Attribution - This research project is not making use of anyone else's work (but then I would say that, wouldn't I?); I will however be referencing authors whose work has influenced my approach.

Standards - As mentioned previously, there are certain issues to consider about potential audiences for sharing. Whilst this blog, my research plan and my reflections are written for an academic audience, I will need to present my results in such a way that non-academic colleagues can benefit from the results. To this end I will be looking for visual ways to present the overall trends of my findings, and different ways of explaining what I believe to be the most relevant theoretical background.

Breaking down the different issues involved in ethics this way has been far more useful to me than having a code of practice laid out in any format. These questions have to be asked time and time again, for every context, to keep them from being ignored. Hopefully they won't just stay on the pages of my Wiki for assessment purposes...

References:

• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.

Happy accidents or inspired designs?

Delving through the literature about online communities, I came across an earlier survey by Johnson (2001), treating online communities of practice as a naturally occurring phenomenon within designed online communities. Johnson studied successful communities of practice, to establish what organisational factors contribute to their success, as opposed to more traditional organisational decision making.

From my worldview, this contrasts the rigid doctrines of hierarchical institutions with the distributed, decentralised approaches of emergent networks. Interesting points concern:

1. The need for organisations to adopt these ways of working to survive in the face of rapid change
2. The concept of collaborative knowledge outweighing individual knowledge - some parallels with collective intelligence perhaps?
3. The tension between the need for the community to understand process, and the need for market & product development

All of these points have particular relevance for the organisation I work for, with rapid changes in business, learning, technology and government. Moving towards this way of working is particularly desirable for transparency and continued development, but is difficult to achieve in practice. My own interest is in understanding - from a qualitative perspective - what experiences encourage people to go beyond occasional participation and into sustained discourse (Hobson, 2013). I am also interested in understanding factors that lead to attrition, a point that Johnson notes as a common problem.

Johnson also addresses issues of the medium itself, our conceptions of community, and the effect of using face-to-face communication alongside online interactions. Effective online communities adhere to adult learning principles, with a breakdown of the normal instructor - learner differentiation. I note that this supports a move from hierarchical to networked organisation principles. Real and sustainable communities will form around real problems where motivation to seek out solutions is high.

Does such collaboration come about within a company naturally? Does it need any facilitation?  What role should learning and development play in all of this? Our role should be to support the development of collective and employee knowledge, rather than to impart it, leading to the moderator role. Here I can identify where my existing efforts have either met with success or struggled. Communities need goals to achieve, objective evaluation of that success [as a group], and also peer & self-evaluation.

There is a need to draw out the concepts of community that apply to work-based problems and those for formal education courses. A key difference is that for formal education, we are all expected to learn the same thing, and are assessed against this, whereas for real workplace learning, it is the variety of individual learning outcomes that allows the group to have greater intelligence than the sum of its parts. It is perhaps this necessary diversity of outcomes that makes online learning communities such a slippery but valuable prize.

Psychology naturally seems to underpin the development (or not) of a community, with perceived intentions having impact on whether learning happens as a result of comments from other members. 'Are they criticising me? How dare they!' or 'It's really helpful to get constructive thoughts on my work' - which of these a person thinks about comments from peers will likely depend on the underlying environment and individual mentalities. It's not simply a question of technology! My next stop for reading will be works by Palloff and Pratt, as mentioned by the author.

References

• Hobson, J. (2012). The problems with educating a quadriform society. Thinking and learning in the digital age [blog] 7 August. Available at: http://jimmyhob.blogspot.co.uk/2012/08/the-problems-with-educating-quadriform.html
• Hobson, J. (2013). Online communities - trying to walk the talk. Thinking and learning in the digital age [blog] 19 March. Available at:  http://jimmyhob.blogspot.com/2013/03/online-communities-trying-to-walk-talk.html
• Johnson, C.M. (2001). A Survey of Current Practice on Online Communities of Practice. Internet and Higher Education, Vol.4 pp.45-60.
• Malone, T. W. (2012). Collective Intelligence. Edge.org [online] 21 November. Available at: http://edge.org/conversation/collective-intelligence