Selection of research method

Draft section of my reflective assignment, based on the blog post ‘A little more action (research) please!’

I have been forced to challenge my own pre-disposition towards quantitative research methods, which was influenced by my physical sciences background. I identified this bias at an early stage in my writing, and found the distinction with qualitative research better defined in my mind by reading the comparisons made by Creswell (2009, Ch.7) and Newby (2010, Ch.3). I also realised that my previous experiences lacked any real involvement in the formal planning of research; my previous projects has always been funded without me having to submit research proposals myself. I decided that qualitative approaches seemed better suited to my context, but it took some time to fully challenge my unconscious habits. I was able to identify possible sources of bias towards theoretical models that I had used (Salmon, 2004).

However it took some additional reading (and re-reading) to fully isolate my unconscious assumptions. After extensive reading about how to create both quantitative and qualitative research proposals, I believed that I had created a set of reasonable questions for qualitative research. My initial research questions were phrased as ‘What effect does...’ and ‘How does...affect...’ Only by revisiting some of the initial reading did I notice that, despite my initial conclusions that qualitative research would be the best approach, that I had automatically designed my questions in a directive way, that would lead to bias towards theory rather than interpreting participants responses from a neutral standpoint.

After some more reading, I was able to present a much more complete and reasoned overview of my research proposal, showing a great deal more thought. This revised research plan fitted much more closely with the principles of action research (Creswell, 2009; Newby, 2010 pp.623-4; O’Brien, 1998), and represents a successful change in my thinking about research.

‘Re-learning means to abolish some toxic assumptions’ (Leonhard, 2013)

References:

• Creswell, J. W. (2009). Research Design: qualitative, quantitative and mixed methods approaches. (3rd edition) Sage.
• Leonhard, G. (2013). Beyond the obvious: re-defining the meaning of learning in a networked society. (video online) Available at: https://www.annotag.tv/learningtechnologies/play/18320
• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.
• O'Brien, R. (1998). An Overview of the Methodological Approach of Action Research. (online) Available at: http://www.web.ca/~robrien/papers/xx%20ar%20final.htm (Accessed March 2013)
• Salmon, G. (2004). e-Moderating: The Key to Teaching and Learning Online. 2nd ed. London: Routledge-Falmer.

A little more action (research) please!

I've been thinking about some of my learning experiences during the course, and I've picked out one of the most critical for getting my qualitative research project on the right tracks.

When I was devising my research questions, the original versions came out like this:

Primary research question:
"What effect does online interaction have on participants' approach to learning?"

Secondary research questions:

1. "How does online socialisation before a face-to-face training event affect interactions within that event?"
2. "How does online interaction before a face-to-face training event affect participants preparation for that event?"
3. "How does the opportunity for online interaction after a training event affect the application of that training in the workplace?"
4. "What would be the effect of allowing participants to contribute anonymous comments to online discourse?"

Fortunately I was reading through the advice given by Creswell (2009, Ch.7) on devising research questions for qualitative research, and I realised that my choice of words was completely inappropriate. Using the word 'affect' (or 'effect') naturally leads towards a more quantitative result because it is inherently directional for responses, as opposed to the exploratory nature of asking people to describe their experiences.

A secondary learning experience that is occurring even as I write this is revision of some of the points I had originally put into my wiki pages. I described my change in questions as being due to advice that Creswell gives specifically about action research, but I've actually misread something again - the advice was simply about qualitative versus quantitative. But I digress...

With a much clearer mind, I can re-define my overall goal for this research project as being to draw out participants' experiences of using online discussion alongside a face-to-face training event, without seeking to establish whether the effects are positive or negative through the questions themselves. So my revised set of research questions comes out as:

Primary research question:
"How would participants describe their experiences of using online interactions to support a face-to-face training event?"

Secondary research questions:
1. "Describe your experience of socialising with other participants who you interacted with online before the training event"
2. "Describe your experience of preparing for a face-to-face event where online interaction was required, relative to an event with no prior interaction"
3. "Describe your experience of participating in online interactions around course related content after the face-to-face event"

I've deliberately left out the question of anonymity - not sure I want to open that can of worms right now!

Reference:
Creswell, J. W. (2009). Research Design: qualitative, quantitative and mixed methods approaches. (3rd edition) Sage.

Evaluating online communities

Thoughts on Ke & Hoadley (2009)

 

Important point to consider: do we expect online learning communities (OLCs) to appear spontaneously or through design? Study inherently seems to favour studies of those that are well-defined in terms of support & structure, so either well-designed or evolved.

Taxonomy of online learning community evaluations

Recognises that this is a divergent research area, attempts to categorise studies in terms of four key components:

1. Evaluation purpose
Notable distinction between proving and improving purposes – convincing organisations that the community has a value at all, versus looking for ways of systematically enhancing the interactions within it. In my case the interactions don't yet amount to a community, so the emphasis is on making interactions sustainable and of benefit to participants.

2. Evaluation approach
Approaches were sometimes summative, usually for proving, or formative for improving, sometimes with elements of both of these. There are also the participatory and responsive approaches – a choice of whether to include participant evaluation or not. Oliver (2000) and Patton (1997) are cited as primary references here. My approach will be based entirely on participant responses, with a view to formative evaluation of the interactions.

3. Measures for evaluation
Outcome vs process measures. The outcome view looks at the community as a static system, evaluating the raw technical set-up of the environment and the learning outcomes. Process evaluation takes an in-depth look at the factors that facilitate or impede learning within the system. My study will need to focus on the process, with a possibility for pairing this up with outcome evaluations from colleagues.

4. Evaluation techniques
The authors seem to use the term objective in place of quantitative; they also refer to qualitative and mixed-method approaches. An important distinction between the two main forms is made – objective approaches deliberately remove context from the data, focusing on what could be directly comparable between other studies. Qualitative studies allowed for more direct insights for the learning processes within a community. I will be focusing on qualitative approaches, although there is potential for identifying the best factors to use in future studies for building up an objective measure of communities in future.

Conclusions

This gives a very good critical analysis of the factors at play in evaluating OLCs, and can serve as a guide point relating to the higher level discussions of Newby (2010), Cresswell (2009) and Colquhoon (2006). The authors also point out a good number of shortcomings in current research. Partly these are due to the constraints of researchers performing their studies for their own purposes, rather than to sit conveniently into the wider body of research. They also point out that the offline interactions between participants play a large role in the actual learning process, and these are very difficult to find any record of. There are also no studies that show how a community has evolved over time.

Long term goals of the researchers are to establish a framework for understanding OLCs, possibly towards a central theory. They make no reference to the five stage model identified by Salmon (2003); they do refer to phases of community development (Palloff and Pratt, 1999). It will be useful to pursue this systematic approach for research to inform my own understanding, and note if any similarities or contradictions with Salmon’s model emerge, which has previously been central to my understanding of online learning interactions.

Reference:

• Colquhoon, D. (2006). Research Methods in Education Contexts. University of Hull.
• Creswell, J. W. (2009). Research Design: qualitative, quantitative and mixed methods approaches. (3rd edition) Sage.
• Ke, F. and Hoadley, C. (2009). Evaluating Online Learning Communities. Educational Technology Research and Development, 57(4), pp.487-510.
• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.
• Salmon, G., 2003. E-Moderating: The Key to Teaching and Learning Online. 2nd ed. London: Routledge-Falmer.

Where, who, what, how and why?

Some questions based on Creswell (2009) and answers...

a) Where is the setting for your research project?

Multiple offices of an educational services provider in the United Kingdom.

b) Who are the people that you will study (or what is the thing that you will study)?

Company employees with responsibility for managing the performance of others, participating in a people management skills programme. Participants come from a range of departments.

I will be studying the experiences that participants have with online interactions which are required before and after the central face-to-face training event.

c) What methods do you plan to use to collect data?

Survey questions of participants...

Central question: How would participants describe their experiences of using online interactions to support a face-to-face training event?

• Describe your experience of socialising with other participants who you interacted with online before the training event
• Describe your experience of preparing for a face-to-face event where online interaction was required, relative to an event with no prior interaction
• Describe your experience of participating in online interactions around course related content after the face-to-face event

d) How will you analyse the data you collect?

Comparison of individual experiences of participants, with a view to emerging patterns

• Across all participants
• Within separate cohorts
• Experiences of socialisation activities
• Experiences of knowledge sharing activities

And my own question...

e) Why am I doing this (besides needing a project for assessment)?

Rapid change in all areas of business brings an increased demand for effective training of employees, with lower budgets and time scales than ever before. Face-to-face training, while still highly valued, is expensive in terms of removing employees from their regular work activities, and the cost for transport and accommodation. There is also a question of how effectively the content of a 'formal' learning intervention of this type can be effectively transferred back to the workplace. Online interactions are of interest as a means for extending and enhancing the learning experience for participants.

References:

• Creswell, J. W. (2009). Research Design: qualitative, quantitative and mixed methods approaches. (3rd edition) Sage.

Thoughts on defining research purpose

Trying to define my research purpose effectively has required extensive reading and critical thought to make sure that my purpose is properly defined

Chapter 5: The Introduction

The Research Problem – Creswell gives some good advice that relates general writing skills to the research context. We are normally trying to convey a very complex topic, so it is important that we consider how we are going to ‘hook’ our readers into the subject, and make sure we don’t bombard them with too much information too soon. There is also the need to make sure that our work is seen as a scholarly endeavour by outlining our links to the literature.

 

Studies Addressing the Problem – Using an inverted triangle seems like a good idea for the process of zeroing in on our research problem, using the wider body of literature to provide the background. We need to acknowledge that there are limitations in the literature we draw on (otherwise there wouldn’t be anything to research) and instead look for ways that the lessons learned or methods employed could inform our own research.

Deficiencies in Past Literature – Whilst avoiding outright condemnation of the existing literature, it is important to point out where it does not (or cannot) address our particular area. For instance, whilst I am planning to use Ke & Hoadley (2009) as a primary reference point for methods, it will be useful to explore the different contexts of the studies they included in their review - how many of the studies are based in the workplace? Referring to studies of learning in the workplace could help to draw links – thinking of the inverted triangle here.

Significance of a Study for Audiences – This does have some good points about writing for particular audiences, and trying to make the study appeal to a wide variety of people. Some of my original reservations remain around the default ‘market’ mentality!  Interestingly one of the sidebars mentions organisational mentalities, and may be worth following up in its own right.

Summary – The five part introduction model should serve well in conjunction with critical thought about my world view and the literature that I will use. The writing exercises will help me to set out my thinking ahead of writing the actual paper, and hopefully draw in some useful feedback. I will make sure that I give feedback to my peers first as an incentive for them to reciprocate, and to strengthen connections amongst the group.

 

Chapter 6: The Purpose Statement

 

Creswell states that this is the most important statement in the entire study – it effectively sets out the means by which others are actually going to be assessing the worth of your research.

 

Significance and Meaning of a Purpose Statement – The purpose statement is deliberately unpicked from the separate points of research problem (the wider issues) and research questions (the fine details of the study).

 

A Qualitative Purpose Statement – Careful use of language is required here. Using the draft script to produce some initial ideas and then going back through the text as a preliminary round of feedback sounds more sensible for learning than trying to construct one from the example words given. That way I can reflect on the differences between how I normally think internally, and the way I externalise this in my writing.

 

A Quantitative Purpose Statement – Although I consider that my study will be more qualitative in nature, it is worth considering the different way that a quantitative purpose statement should be written, so that I know the difference! Considering the variables that may affect an outcome and systematically investigating them is a hallmark of a quantitative study. Although I need to consider what factors may affect communities, I will not try to quantify them at this stage. Theory is implied as being more central to a quantitative study than a qualitative one.

 

A Mixed Method Purpose Statement – Understanding the previous two types of purpose is vital for properly introducing a mixed methods study, as the reader has to be able to distinguish between them through your writing. For success, there should be a central theory, and a single qualitative phenomenon under discussion.

 

Summary – Universal techniques underpin all of the types of study. Action words must be used for outlining the purpose, whilst keeping to non-directional language (i.e. avoid trying to prove a hypothesis through interfering with data acquisition). Mentioning the strategy for the enquiry, along with the participants and site to provide context, helps a reader further discern the relevance for them.

 

Influences on the selection of a research issue (Creswell)

Research agenda – Ultimately everything needs to be paid for! Reading through some of Creswell’s assumptions on the last unit led me to challenge the assumption that research has to be targeted at the greatest number of people. This actually fits more with a market philosophy, whereas I would rather that my research fitted into the network paradigm, and also be carried out for on-going practitioner development. My organisation wants more learning to go online, but there are likely to be many different perceptions of what does and does not work. I may actually be going against accepted wisdom with my desire to research this area!

 

Research issue – From my context, online communication actually becomes a potential measure for how successful a traditional learning event has been, as opposed to one where online interactions have not been present. The Research Problem can then be further defined as outlining effective ways of evaluating training success beyond the event, fitting in with the research agenda more closely.

 

Research question – Can online learning interactions increase the effectiveness of training programs, and when are they most likely to succeed?

 

Learning from a research agenda - This section gives some very good insights on how the research agenda can shift, and acknowledges the limitations of academic journals in our learning and research.

Determining a research issue - Generation of ideas, particularly questions, is dependent on having a rich and diverse environment.  In this way we are more likely to see alternative perspectives that allow us to challenge agendas and assumptions.

Kick-starting the search for a research issue - Whilst this is not the first research project that I have conducted, I do think that I am better regarding myself as being a ‘new’ researcher as I did not undertake any of this kind of learning before my research project – everything was already funded before I started! Within my department there is not always formal research, although my colleagues did use the Delphi research method for determining the best approach to performance & talent management. I also have the benefit of having colleagues who work in educational research to connect with.

 

Scoping the research problem - There are some interesting thoughts here. The reference to Mallik and McGowan about workplace learning is potentially very helpful, although some care will be needed for drawing similarities. How feasible any research will be in my area will sometimes depend on people’s willingness to provide feedback, which is not always guaranteed as people are busy in the workplace. I will have to be careful to make use of different methodologies, as I will have to continue researching towards my dissertation.

References:

• Creswell, J. W. (2009). Research Design: qualitative, quantitative and mixed methods approaches. (3rd edition) Sage.
• Ke, F. and Hoadley, C. (2009). Evaluating Online Learning Communities. Educational Technology Research and Development, 57(4), pp.487-510.
• Newby, P. (2010). Research Methods for Education. Pearson.