The 'Carry On' factor

Image: freeimages

Following on from my thoughts about quantitative research, I'm looking at some of the dependent variables that will come into play, and thinking about how I might go about analysing them.

Intention to continue examining & job satisfaction

This is an extremely important factor for exam boards, as they are dependent on a large network of examiners to make our examinations possible. Meadows (2004) identified four factors that affect examiners' attitudes towards their jobs:

1. The pressure and stress of examining
2. Insight gained from examining
3. Support from awarding body and senior examiners
4. Pay

However, Meadows found that only the pressure and stress of examining, and the level of support received, predicted intention to continue examining; however pay did affect examiners' job satisfaction. One of the key sources of stress came from balancing examining duties with regular work, with the report recommending that resources should be diverted to lobbying for examiners to be given more time away from teaching to examine, in order to improve retention. Improving the level of support was also a recommendation, although the report notes that this would be less cost effective, since most examiners were already relatively satisfied with the support they received. Increasing pay would improve job satisfaction, but the report states that this would not improve retention.

The introduction of software tools

Tremain (2011) followed up this work to consider how the situation had changed after the introduction of electronic marking and online standardisation. The study looked at the factors that influence the satisfaction that examiners express about their work, and highlighted three factors underpinning examiners' intention to continue:

1. The relationship between examining work and work outside examining
2. The pressures of examining and support received
3. The incentives for examining

The study states that although there is no imminent threat to examiner retention, future threats include the increasing use of online tools, which can contribute to examiners feeling unsupported or undervalued. Job satisfaction is considered to be more important in retention than reward for the majority of jobs, with social interaction and appropriate challenge being considered particularly valuable. The adoption of online tools had contributed to a sense of isolation amongst examiners, and also made the work more routine - although the reliability of marking has actually increased as a result.

A further study (Tremain, 2012) also set out to evaluate how specific factors involved in online marking & standardisation contributed to examiner satisfaction. This concluded that there was no significant difference in intention to continue marking between examiners who were standardised using face-to-face or online methods. Examiners who had marked using a mixture of paper and online methods showed a very slight increase in intention to continue examining. However, it was noted that the results were confounded by the different subjects and levels of experience amongst the participants.

Variables that we may be able to influence, and how:

• Support received. By considering the different levels of support that are currently offered from the contextual model for learning (Shepherd, 2011) and identifying possible gaps, we may be able to improve the support offering for examiners in a rational way. I have already laid out some initial thoughts for this approach.
• Insight gained from examining. Making key insights from senior examiners available in a digital form which can be shared more easily online, for instance through learning management systems or webinars, could help to ensure efficient dissemination of relevant information.
• Social interaction. This is a long term goal that our organisation may want to consider for retaining examiners. Although we are increasingly unable to provide opportunities for examiners to meet in a face-to-face setting, there are possibilities for facilitating some more informal interaction around scheduled events. One of my colleagues is keen to run webinars for examiners to gain insight from senior examiners, and careful use of online chat could help to provide a better sense of community.

Any or all of these methods could be attempted, with measurement of the effect on intention to continue, and also examiner performance, being undertaken to determine effectiveness. One concern I have is that apparent failure to make a difference at first might result in a loss of enthusiasm for innovation, hence there would need to be trust established with stakeholders for future improvements. Undertaking action research alongside quantitative measurements to demonstrate a rational approach would be key to successful establishment of such trust.

References:

• Meadows, M., 2004. Measures of examiner job satisfaction as predictors of intention to continue examining. Available at: https://cerp.aqa.org.uk/research-library/measures-examiner-job-satisfaction-predictors-intention-continue-examining [Accessed December 8, 2013].
• Shepherd, C., 2011. The new learning architect, Chesterfield, U.K.: Onlignment.
• Tremain, K., 2011. Carry on examining: what predicts examiners’ intentions to continue examining? Available at: https://cerp.aqa.org.uk/research-library/carry-examining-what-predicts-examiners-intentions-continue-examining [Accessed December 7, 2013].
• Tremain, K., 2012. Carry on examining: further investigation. Available at: https://cerp.aqa.org.uk/research-library/carry-examining-further-investigation [Accessed December 7, 2013].

Architecture

Going off on a little bit of a tangent, it's time to take a look at what support is being offered to our examiners through the e-learning provision, and how it could be improved. Shepherd (2011) provides a contextual model for learning, based on four contexts: experiential, on-demand, non-formal, and formal; and two perspectives: top-down and bottom-up.

• Experiential learning is learning from as opposed to learning to. We have to be actively engaged with our task and - one hopes - reflecting on our successes and failures. There are a great wealth of lessons that can be learned while examining - Meadows (2004) and Tremain (2011) report that examiners consistently cite the insight gained from examining as one of the key benefits. Considering how to support or encourage such reflections might help to improve examiner performance.
• On-demand learning is learning to perform a particular task and acquire the necessary knowledge, at the point of need or 'just-in-time'. Depending on how far in advance our examiners access the learning materials, they could be regarded in this fashion, although they are probably better used in the following category.
• Non-formal learning is also learning to but with a more relaxed time frame, where employers take steps for employees to be prepared in the medium to long term, and is sometimes labelled as 'just-in-case' learning to set it apart from on-demand. It is distinguished from formal learning below by not being packaged as a formal 'course', which will be the case for this intervention, although there may be something to be said for carefully considering how the materials are to be presented to examiners.
• Formal learning is defined by clearly set learning objectives, a means of assessment, and usually some kind of qualification. We definitely don't offer a qualification for learning to examine (perhaps some would say we should?!), assessment would be somewhat laborious unless it were to be done covertly through completion, and the learning objectives are difficult to define. It is far easier here to think in terms of business objectives - 'There are no learning metrics, only business metrics' (Cross) That being said, it will be worth considering what examiners will expect to see and make sure that objectives are clearly stated.
• Top-down learning is aligned with employers' objectives, and is intended to ensure performance is not left to chance, and sets out to ensure that the requisite skills and knowledge can be acquired to do so. Our organisation is responsible for ensuring that results are delivered on time and accurately, with severe penalties possible for failing to do so.
• Bottom-up learning occurs because of employees' motivation to perform effectively. In addition to the motives around improved insight, examiners are drawn to the extra pay for examining, and improved promotion prospects in their teaching roles (Meadows, 2004).

Moving into specifics from Shepherd's model, there are several components that are either currently present in our business model, introduced explicitly through the e-learning provision, and some that perhaps should be there. The existing provisions focus exclusively on the top-down perspective:

• There are already performance appraisals built into our way of working (experiential);
• A help-desk is provided through our Contact Centre (on-demand);
• Examiners typically receive a certain amount of on-job training through contact with their supervisors (non-formal);
• For the live pilot, examiners will be receiving classroom courses (formal);
• The majority of users will have access to the rapid e-learning materials (non-formal). Note that I avoid referring to our e-learning as self-study e-learning (a formal intervention), which by definition should provide 'instant and individualised feedback', something that is far beyond the scope of our planning.

If we open ourselves up to the full range of possibilities from Shepherd's model, I would be tempted to add the following methods:

• Webinars could be used to convey a lot of the material and briefing that might take place in a face-to-face context, without examiners having to travel to a central location, and allow for some questions & answers
• Online video could be a powerful tool for engaging examiners with the task at hand, especially if delivered by senior examiners involved in the pilot. The message would have to be particularly clear, relevant and to-the-point, requiring serious consideration before asking for this intervention.
• Performance support materials could be leveraged for contact centre staff and senior examiners, drawing on key lessons from the live pilot.
• Forums could be used amongst contact centre staff to post common questions from examiners; there could also be forums available to examiners to make common or emergent solutions available.
• Of course I would love it if someone other than myself found a reason to keep a blog ...

That's all for now I think, I'll look at specific applications in future posts....

References:

• Meadows, M., 2004. Measures of examiner job satisfaction as predictors of intention to continue examining. Available at: https://cerp.aqa.org.uk/research-library/measures-examiner-job-satisfaction-predictors-intention-continue-examining [Accessed December 8, 2013].
• Shepherd, C., 2011. The new learning architect, Chesterfield, U.K.: Onlignment.
• Tremain, K., 2011. Carry on examining: what predicts examiners’ intentions to continue examining? Available at: https://cerp.aqa.org.uk/research-library/carry-examining-what-predicts-examiners-intentions-continue-examining [Accessed December 7, 2013].

Time for a numbers game?

Image: freeimages

Following on from my last blog post, I'm re-treading the sequence of reading from our Research Methods module to get my bearings again, and I'm coming back to the question of qualitative vs quantitative research. While I strongly identified with the Action Research methodology on my last project, it's worth deliberately opening up my mind to new possibilities, especially as there will be strong interest in some kind of numerical data from colleagues and external auditors if we are questioned on our approach.

So before I start to choose which quantitative disciplines I might wish to draw on, I'll look at the key aspects of quantitative research, consider those that appeal to me, and those I wish to avoid.


Concern with theory

Relating my findings to theory will be helpful to ensure some kind of tethers to related work, but there's a danger of getting obsessed with reproducibility and control here. When you're moving into the realm of on-demand learning, you can't guarantee learning outcomes, nor indeed that learners will even access the materials or activities that you produce for them. Newby (2010, p.96) acknowledge the limitations for educational researchers trying to identify pattern and control influences, as they are only able to view a small part of the overall education system. I would prefer to think in terms of Praxis (Wheeler, 2013), which requires practitioners to consider how closely their practice overlaps with the theories they identify with.

Concern with proof

Here lies one of the real problems for educational research - although I understand that establishing proof would give greater peace of mind, the complexity and ambiguity of the situation makes this extremely difficult:

• The situation I face will not be the same as another practitioner does, even if our verbal descriptions of it seem similar to the untrained eye
• The next situation that I (and the learners) face will not be the same as this one, even if it's 'just another e-marking system'
• The time needed to establish proof would be completely at odds with the time pressures for the project, where the learning is 'on-demand'.

The best that I can hope for is to show that using theories to guide my design leads to a dependable business outcome, and that particular methods or techniques are better suited to my situation.

Identification of variables

This is one of the key aspects of quantitative theory that I see as helpful. Although my control over most independent variables involved will be limited to say the least, it will definitely be helpful to at least make some systematic efforts to identify variables in the design of materials that may be having an effect, and to measure any dependent variables which are of interest. Our particular concerns would be the performance of examiners, and intention to continue based on their experiences. Attempting to correlate these with participation in the different aspects of the support might yield useful insights into which components have succeeded, but this would have to be linked to effective practice in design.

Simply saying that an approach should be abandoned because it doesn't seem to have an effect in this situation would be potentially misleading without some understanding as to why. Creswell (2009, p.49) refers to confounding variables (e.g. discriminatory attitudes) that can come into play, which I have had some experience of when trying to introduce online learning methods in the past. Participants who are negative about the use of the tools go to great lengths to discredit them when given the opportunity to do so, whilst the majority of participants actually acknowledge a positive effect.

Conclusion

This project will benefit from the use of some quantitative approaches to analysing data about examiner performance and intention to continue, but these will need to be paired effectively with qualitative methods to understand what dependent variables relating to the choice and design of approaches might be influencing the outcomes. My next blog post will focus on the type(s) of quantitative research methods might be useful, followed by a look at rational design approaches for the learning provision.

References:

• Creswell, J. W. (2009). Research Design: qualitative, quantitative and mixed methods approaches. (3rd edition) Sage.
• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.
• Wheeler, S. (2013). Praxis makes perfect. Learning with e's [blog] 31 October. Available at: <http://steve-wheeler.blogspot.co.uk/2013/10/praxis-makes-perfect.html>

Back to where it all began

Image: freeimages

After a year or so of working on staff development programmes and doing a lot of creative and novel work, I'm back to where my career in online learning really began: making tutorial videos and briefings for examiners. As always, the time pressure is intense and I'm largely working solo - while some might see this as stressful I'm actually looking forward to it, because it's a welcome chance to really reflect on the circumstances where I first honed my skill set.

'Experience: that most brutal of teachers. But you learn, my God do you learn.'

C. S. Lewis

The tools changed several times in the first few years. The first recording software we started using because we had already invested in related software - I won't specify which software here, because it may have moved on since. You could record your screen with live narration, it was possible to easily publish the material using a hyperlink, the results could be reasonable, but it wasn't without its flaws. Particularly the editing could be a real nightmare if you made any mistakes in your recording, and quality seemed to degrade with each edit.

The second time around we started using Techsmith Camtasia Studio, which was a massive revelation. Screen recording was a great deal sharper, the editing process was vastly improved and I quickly found the value of highlighting, captions, and zooming & panning the view to draw attention to relevant areas. We now had the freedom to publish to good quality video formats, and with backup from the web team we could publish videos to an orphan page for examiners to view.

Finally we moved on to Adobe Captivate, which we have stuck with ever since for software demonstrations. It's a lot more technical than other software, which may put some people off, but it's allowed me to move forward with creating more interactive material (particularly simulations). As we finally moved over to our own LMS, the software had what we needed to publish with all the e-learning information for SCORM packages.

I've learned to stay mindful of the advice from Henderson (2012) to avoid being trapped by the tools, and that of Toth (2012) to choose the right tool for the job, so I always look for opportunities to use different e-learning tools. However I am finding it harder to take on new tools as my time gets increasingly bound up in development, so perhaps now isn't the time to take on something new for the recording. Instead I'll be looking to draw on the advice from Shepherd (2011) around considering carefully the context of your learners and what support they might need. In subsequent posts I'll be drawing up outlines for the additional approaches that might be used, and the opportunities to draw in different tools.

I'm automatically thinking of using Action Research methodology, since it was successful for last project, but this talk of not being trapped by your tools has made me pause. Perhaps it's worth re-treading some of the exercises from the Research Methods course and make sure that Action Research, and indeed qualitative research, is the correct approach.

References:

• Henderson, A., 2012. Don't get trapped by your e-learning tools. In: Allen, M.W., 2012. Michael Allen’s E-learning annual 2012, San Francisco, Calif.: Pfeiffer.
• Toth, T.A., 2012. The right e-learning tool for the job. In: Allen, M.W., 2012. Michael Allen’s E-learning annual 2012, San Francisco, Calif.: Pfeiffer.
• Shepherd, C., 2011. The new learning architect, Chesterfield, U.K.: Onlignment.

Selection of research method

Draft section of my reflective assignment, based on the blog post ‘A little more action (research) please!’

I have been forced to challenge my own pre-disposition towards quantitative research methods, which was influenced by my physical sciences background. I identified this bias at an early stage in my writing, and found the distinction with qualitative research better defined in my mind by reading the comparisons made by Creswell (2009, Ch.7) and Newby (2010, Ch.3). I also realised that my previous experiences lacked any real involvement in the formal planning of research; my previous projects has always been funded without me having to submit research proposals myself. I decided that qualitative approaches seemed better suited to my context, but it took some time to fully challenge my unconscious habits. I was able to identify possible sources of bias towards theoretical models that I had used (Salmon, 2004).

However it took some additional reading (and re-reading) to fully isolate my unconscious assumptions. After extensive reading about how to create both quantitative and qualitative research proposals, I believed that I had created a set of reasonable questions for qualitative research. My initial research questions were phrased as ‘What effect does...’ and ‘How does...affect...’ Only by revisiting some of the initial reading did I notice that, despite my initial conclusions that qualitative research would be the best approach, that I had automatically designed my questions in a directive way, that would lead to bias towards theory rather than interpreting participants responses from a neutral standpoint.

After some more reading, I was able to present a much more complete and reasoned overview of my research proposal, showing a great deal more thought. This revised research plan fitted much more closely with the principles of action research (Creswell, 2009; Newby, 2010 pp.623-4; O’Brien, 1998), and represents a successful change in my thinking about research.

‘Re-learning means to abolish some toxic assumptions’ (Leonhard, 2013)

References:

• Creswell, J. W. (2009). Research Design: qualitative, quantitative and mixed methods approaches. (3rd edition) Sage.
• Leonhard, G. (2013). Beyond the obvious: re-defining the meaning of learning in a networked society. (video online) Available at: https://www.annotag.tv/learningtechnologies/play/18320
• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.
• O'Brien, R. (1998). An Overview of the Methodological Approach of Action Research. (online) Available at: http://www.web.ca/~robrien/papers/xx%20ar%20final.htm (Accessed March 2013)
• Salmon, G. (2004). e-Moderating: The Key to Teaching and Learning Online. 2nd ed. London: Routledge-Falmer.

Reliability

Is 'reliability' the right word to use for qualitative research? Cohen et al. (2007, Ch.10) suggest a variety of different words that should be used to distinguish the variable-controlled nature of quantitative research from the variable-embracing nature of qualitative research. Points addressing reliability that are present in my research include repeating the method of data collection and analysis across several groups (related to triangulation, below) and using anonymous questionnaires to encourage honesty amongst participants (p.209).

Early on in my research plan I discussed some of the principles laid out by Newby (2010, pp.120-3) for ensuring some degree of reliability in qualitative research, particularly the concept of triangulation. In the case of my project, the point about getting information form multiple sources is the best source of reliability. The questionnaire has been answered by participants from different cohorts of the same program, so we can gain some idea of how reliable (or not) the results are by comparing different cohort responses. Although the research is qualitative in nature, the reliability can be commented on by comparing the number of responses with the actual number of participants for each cohort.

References

• Cohen, L., Manion, L., Morrison, K. & Bell, R. (2007). Research methods in education. Routledge Limited.
• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.

Data analysis

Although I set out to do a qualitative research project, I did reserve the right to quantify my results to some extent. I've found it helpful when making sense of my data to do a quick finger-test of whether comments were positive, neutral or negative, and summarise them by question and group. These can then be totalled and presented graphically, which will be very helpful when sharing back with participants. The idea of sharing with participants as a first audience seems particularly prudent to avoid accusations of 'intellectual imperialism' (Newby, 2010, p.48). This will also act as an important feedback exercise to help practitioners see the reaction of participants, and to give me more direct contact with both parties to better understand how to inductively expand the research.

There is still a lot more work to do on full data analysis at this stage, but deadlines are looming!

References:

• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.

Collecting the data

I've been a bit lax in following the last stages of the pathway, so I thought I had better get some thoughts down before they fade away. 

What challenges do you face and what successes do you have?

Challenges:

• Some of the experiences I am asking people to recount are in some cases from over 2 years ago.
• The return rate for the questionnaire has been around 50% of all possible participants - partly this is due to it being a busy time of year for staff.

Successes:

• I have received a good number of responses, with an excellent level of detail
• Responses have yielded insights into how to improve the processes in question

What unexpected experiences do you have?

The use of an online questionnaire has left me feeling somewhat disembodied - most of the methods described in our reading concerned face-to-face methods. However they have been very effective, and in keeping with the idea that online learning should be anytime, anywhere...

How does your experience of data collection reflect, or conflict with, the principles and theories encountered in your reading?

I realise from reading through Newby (2010, pp.332-3) that I didn't pilot the questionnaire as rigorously as I should have done.  My testing was limited to sending the link to one participant who I asked about trialling it beforehand. Fortunately I had made significant revisions to my questions early on in the project that kept the research on track.

References:

• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.

The nature of academic publishing

Image: forwardcom via freeimages

Building up towards my reflective assignment, I thought it might help to draft this in sections, according to particular areas of interest throughout the module. First stop is publishing...

One of the assumptions that I am interested in challenging is the area of academic publishing. We tend to regard the world of academic journals as ‘an open and public forum’ (Colquhoon, 2006) that is created for the common good, but a growing number of people are beginning to question if this holds true, especially in the age of open web content. Wheeler (2011a, b, c) has written a number of blog posts criticising the traditional system of academic publishing, noting that:

‘It’s a hierarchy that rarely changes.’

Furthermore, the hierarchical nature of the publishing world is often self-defeating in terms of securing the widest possible audience:

‘...many of the top, elite journals we are expected to publish in are in fact read by a very small percentage of the community the research is intended to reach.’

From considering my own worldview, I have noted the restrictions that hierarchical institutions place on the education system, particularly when combined with the solutions offered by market organisations (Hobson, 2012; Ronfeldt, 2012b). Potential disruptive alternatives may yet be found in the emerging network sector predicted by Ronfeldt (2012a), or the peer-to-peer ideology (Bauwens, 2009), although sustainable business models are yet to emerge. It should be noted that there are dangers inherent in the misuse of data for either solution. Newby (2010) warns of ‘intellectual imperialism’ when researchers are able to claim sole ownership of research data, discounting the rights of those involved in the study. At the other extreme of open web publishing, Lanier (2006a, b; 2012) warns of the dangers of ‘digital Maoism’, whereby all ownership and authorship of published data is lost, and transferred to control of the crowd.

References:

• Bauwens, M. (2009). Open Process academic publishing: towards the peer production of academic peer review. P2P Foundation's blog [blog] 15 September. Available at: <http://blog.p2pfoundation.net/open-process-academic-publishing-towards-the-peer-production-of-academic-peer-review/2009/09/15>
• Colquhoon, D. (2006). Research Methods in Education Contexts. University of Hull
• Hobson, J. (2012). The problems with educating a quadriform society. Thinking and learning in the digital age [blog] 7 August. Available at: <http://jimmyhob.blogspot.co.uk/2012/08/the-problems-with-educating-quadriform.html>
• Lanier, J. (2006a). Digital Maoism: The hazards of the new online collectivism. Edge.org [online] 30 May. Available at: <http://www.edge.org/3rd_culture/lanier06/lanier06_index.html>
• Lanier, J. (2006b). Beware the online collective. Edge.org [online] 25 December. Available at: <http://www.edge.org/3rd_culture/lanier06/lanier06.2_index.html>
• Lanier, J. (2012). Are we at the beginning of the rise of post human machine intelligence? [Video online] Available at: <http://dpcloud.co/v3/enterprise/learningtech/index.php?option=com_dpcloud&v=play&p=7117&u=learningtech&cat=Keynotes&Itemid=2>
• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.
• Ronfeldt, D. (2012a). TIMN in 20 minutes: a video presentation. Visions from two theories [blog] 23 May. Available at: <http://twotheories.blogspot.co.uk/2012_05_01_archive.html>
• Ronfledt, D. (2012b). Q’s & A’s about “TIMN in 20 minutes” (4th of 7): creation of a new sector? Visions from two theories [blog] 29 August. Available at: <http://twotheories.blogspot.co.uk/2012/10/qs-as-about-timn-in-20-minutes-4th-of-7_29.html>
• Wheeler, S. (2011a). Sharp practice. Learning with e's [blog] 26 September. Available at: <http://steve-wheeler.blogspot.co.uk/2011/09/sharp-practice.html>
• Wheeler, S. (2011b). The open case. Learning with e's [blog] 2 October. Available at: <http://steve-wheeler.blogspot.co.uk/2011/10/open-case.html>
• Wheeler, S. (2011c). When the dam breaks… Learning with e's [blog] 18 October. Available at: <http://steve-wheeler.blogspot.co.uk/2011/10/when-dam-breaks.html>

Validity

There is a danger of intertwining the concepts of reliability and validity, so I'm attempting to address them in separate posts.  Cohen, Manion & Morrison (2013, pp. 177-99) devote a great deal of attention to the concept of validity, drawing on a variety of sources to lend weight and richness to the discussion.  Several of the points (p.180) raised are applicable to my research data:

• The natural setting is the principle source of data
• Context-boundedness and 'thick description'
• Data are socially situated
• The researcher is part of the researched world
• The researcher - rather than the research tool - is the key instrument of research
• The data are descriptive
• There is a concern for processes rather than simply with outcomes (inherent in my choice of action research)
• Data are analysed inductively rather than using a priori categories
• Data are presented in terms of the respondents rather than researchers

There are some points to beware of that might undermine the validity of my research - I will also list how these can be addressed:

• Reactivity (internal validity) - I am aiming to improve the processes involved in the learning programs; this should not affect the data for previous cohorts, although the current cohort could potentially be affected by knowing that they are being observed.  This also touches on the issue of Researcher bias mentioned by the authors
• Concensual validity (external) - since I am undertaking this research as a learning experience, will 'competent others' dismiss my findings due to my inexperience, or because they think I'm simply forcing the data to fit so that I pass my assessment?

Triangulation is dealt with as a means of ensuring validity.  Newby (2010, pp. 121-3) also mentions this technique, although in somewhat less detail.  So to what extent do my research methods yield themselves to this?

• Time triangulation - I have used the same method for a number of cohorts, although this effect may be diminished because I did not carry out the survey for each group immediately following their participation, so earlier groups may not recall their experiences as accurately.
• Theoretical triangulation - since I have avoided basing my questions on one particular theory, there is the opportunity to compare the results from the point of view of competing theories for social and online learning.
• Investigator triangulation - the data are recorded electronically, so potentially other researchers could give their own interpretations.
• Methodological triangulation - the same method has been used on different groups, so I can easily compare the results of each group to consider how well the results support conclusions for each group.

General points that have contributed to validity on this project:

• Choosing an appropriate methodology for answering the research question - action research.  This has ensured that the focus is on processes - not outcomes, which I might be interested in unfairly interpreting!
• Selecting appropriate instrumentation - using online questionnaires allowed the data to be gathered according to the time needs of respondents, and allowing them access to reminders (their forum postings) whilst responding to questions.

Limitations here include:

• Sampling - by not requiring participation in the survey, I sacrificed control over sample sizes, which could potentially limit the validity (and reliability) of results.  However I considered that sensitivity to participants' wishes was of greater importance in this case, since some people are uncomfortable with the use of online forums in the first place.

Overall I believe I have sufficiently addressed issues of validity, but there are clearly many others that I have not encountered yet, including the points concerning data interpretation.

References

• Cohen, L., Manion, L. & Morrison, K. (2007) Research Methods in Education.
• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.

Revisiting ethics

Some more thoughts on the ethics of my project, based on the principles outline by Newby (2010, pp.47-9)

Moral position - None of the groups involved in the study will be treated differently, since the actual participation in learning programs has already taken place. However the responses will help to guide myself and my colleagues to ensure that the use of forums is not a detriment to future learning experiences, and ensure that problems experienced by individuals can be addressed.

Honesty - I am not aiming to fit the data to a particular theory or framework from the outset, although I will have to be careful to avoid the temptation to do this. In order to provide real justification for this, such results would have to be reliable and reproducible across several data sets, which is not likely to be possible across such small samples. By choosing action research I am committing to developing understanding of the online environment for practitioners & future participants alike.

Responsibility - All information shared is within the same organisation, and responses are anonymous, with the option to avoid being quoted in written reports, so nobody should be disadvantaged as a result of this study. There are no questions that I would expect to reveal anything sensitive, however this is something to bear in mind for future studies.

Sharing - I have expressed my intent to share results with participants, and with the wider organisation. What I will need to consider is the appropriate format to write up my results for others to appreciate. I'm sure most colleagues won't get much benefit from an academic-style paper! Nor should I be disseminating detailed results outside the organisation, although there is probably some benefit in talking about the overall process on a wider platform - this blog to be precise!

Freedom - Participants have had freedom to answer the questionnaire or not, and have also been given the option to have their responses omitted from written reports. I have not based the study on their actual forum entries, and I would be inclined to avoid doing so at such an early stage in the use of forums. People are apprehensive enough about engaging with the medium without the idea of their every word being used in evidence!

Community - This research project has been discussed with colleagues in Learning & Development to ensure that the nature of questions asked is appropriate, and that participants can benefit from participating.

Attribution - This research project is not making use of anyone else's work (but then I would say that, wouldn't I?); I will however be referencing authors whose work has influenced my approach.

Standards - As mentioned previously, there are certain issues to consider about potential audiences for sharing. Whilst this blog, my research plan and my reflections are written for an academic audience, I will need to present my results in such a way that non-academic colleagues can benefit from the results. To this end I will be looking for visual ways to present the overall trends of my findings, and different ways of explaining what I believe to be the most relevant theoretical background.

Breaking down the different issues involved in ethics this way has been far more useful to me than having a code of practice laid out in any format. These questions have to be asked time and time again, for every context, to keep them from being ignored. Hopefully they won't just stay on the pages of my Wiki for assessment purposes...

References:

• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.

Happy accidents or inspired designs?

Delving through the literature about online communities, I came across an earlier survey by Johnson (2001), treating online communities of practice as a naturally occurring phenomenon within designed online communities. Johnson studied successful communities of practice, to establish what organisational factors contribute to their success, as opposed to more traditional organisational decision making.

From my worldview, this contrasts the rigid doctrines of hierarchical institutions with the distributed, decentralised approaches of emergent networks. Interesting points concern:

1. The need for organisations to adopt these ways of working to survive in the face of rapid change
2. The concept of collaborative knowledge outweighing individual knowledge - some parallels with collective intelligence perhaps?
3. The tension between the need for the community to understand process, and the need for market & product development

All of these points have particular relevance for the organisation I work for, with rapid changes in business, learning, technology and government. Moving towards this way of working is particularly desirable for transparency and continued development, but is difficult to achieve in practice. My own interest is in understanding - from a qualitative perspective - what experiences encourage people to go beyond occasional participation and into sustained discourse (Hobson, 2013). I am also interested in understanding factors that lead to attrition, a point that Johnson notes as a common problem.

Johnson also addresses issues of the medium itself, our conceptions of community, and the effect of using face-to-face communication alongside online interactions. Effective online communities adhere to adult learning principles, with a breakdown of the normal instructor - learner differentiation. I note that this supports a move from hierarchical to networked organisation principles. Real and sustainable communities will form around real problems where motivation to seek out solutions is high.

Does such collaboration come about within a company naturally? Does it need any facilitation?  What role should learning and development play in all of this? Our role should be to support the development of collective and employee knowledge, rather than to impart it, leading to the moderator role. Here I can identify where my existing efforts have either met with success or struggled. Communities need goals to achieve, objective evaluation of that success [as a group], and also peer & self-evaluation.

There is a need to draw out the concepts of community that apply to work-based problems and those for formal education courses. A key difference is that for formal education, we are all expected to learn the same thing, and are assessed against this, whereas for real workplace learning, it is the variety of individual learning outcomes that allows the group to have greater intelligence than the sum of its parts. It is perhaps this necessary diversity of outcomes that makes online learning communities such a slippery but valuable prize.

Psychology naturally seems to underpin the development (or not) of a community, with perceived intentions having impact on whether learning happens as a result of comments from other members. 'Are they criticising me? How dare they!' or 'It's really helpful to get constructive thoughts on my work' - which of these a person thinks about comments from peers will likely depend on the underlying environment and individual mentalities. It's not simply a question of technology! My next stop for reading will be works by Palloff and Pratt, as mentioned by the author.

References

• Hobson, J. (2012). The problems with educating a quadriform society. Thinking and learning in the digital age [blog] 7 August. Available at: http://jimmyhob.blogspot.co.uk/2012/08/the-problems-with-educating-quadriform.html
• Hobson, J. (2013). Online communities - trying to walk the talk. Thinking and learning in the digital age [blog] 19 March. Available at:  http://jimmyhob.blogspot.com/2013/03/online-communities-trying-to-walk-talk.html
• Johnson, C.M. (2001). A Survey of Current Practice on Online Communities of Practice. Internet and Higher Education, Vol.4 pp.45-60.
• Malone, T. W. (2012). Collective Intelligence. Edge.org [online] 21 November. Available at: http://edge.org/conversation/collective-intelligence

A little more action (research) please!

I've been thinking about some of my learning experiences during the course, and I've picked out one of the most critical for getting my qualitative research project on the right tracks.

When I was devising my research questions, the original versions came out like this:

Primary research question:
"What effect does online interaction have on participants' approach to learning?"

Secondary research questions:

1. "How does online socialisation before a face-to-face training event affect interactions within that event?"
2. "How does online interaction before a face-to-face training event affect participants preparation for that event?"
3. "How does the opportunity for online interaction after a training event affect the application of that training in the workplace?"
4. "What would be the effect of allowing participants to contribute anonymous comments to online discourse?"

Fortunately I was reading through the advice given by Creswell (2009, Ch.7) on devising research questions for qualitative research, and I realised that my choice of words was completely inappropriate. Using the word 'affect' (or 'effect') naturally leads towards a more quantitative result because it is inherently directional for responses, as opposed to the exploratory nature of asking people to describe their experiences.

A secondary learning experience that is occurring even as I write this is revision of some of the points I had originally put into my wiki pages. I described my change in questions as being due to advice that Creswell gives specifically about action research, but I've actually misread something again - the advice was simply about qualitative versus quantitative. But I digress...

With a much clearer mind, I can re-define my overall goal for this research project as being to draw out participants' experiences of using online discussion alongside a face-to-face training event, without seeking to establish whether the effects are positive or negative through the questions themselves. So my revised set of research questions comes out as:

Primary research question:
"How would participants describe their experiences of using online interactions to support a face-to-face training event?"

Secondary research questions:
1. "Describe your experience of socialising with other participants who you interacted with online before the training event"
2. "Describe your experience of preparing for a face-to-face event where online interaction was required, relative to an event with no prior interaction"
3. "Describe your experience of participating in online interactions around course related content after the face-to-face event"

I've deliberately left out the question of anonymity - not sure I want to open that can of worms right now!

Reference:
Creswell, J. W. (2009). Research Design: qualitative, quantitative and mixed methods approaches. (3rd edition) Sage.

Evaluating online communities

Thoughts on Ke & Hoadley (2009)

 

Important point to consider: do we expect online learning communities (OLCs) to appear spontaneously or through design? Study inherently seems to favour studies of those that are well-defined in terms of support & structure, so either well-designed or evolved.

Taxonomy of online learning community evaluations

Recognises that this is a divergent research area, attempts to categorise studies in terms of four key components:

1. Evaluation purpose
Notable distinction between proving and improving purposes – convincing organisations that the community has a value at all, versus looking for ways of systematically enhancing the interactions within it. In my case the interactions don't yet amount to a community, so the emphasis is on making interactions sustainable and of benefit to participants.

2. Evaluation approach
Approaches were sometimes summative, usually for proving, or formative for improving, sometimes with elements of both of these. There are also the participatory and responsive approaches – a choice of whether to include participant evaluation or not. Oliver (2000) and Patton (1997) are cited as primary references here. My approach will be based entirely on participant responses, with a view to formative evaluation of the interactions.

3. Measures for evaluation
Outcome vs process measures. The outcome view looks at the community as a static system, evaluating the raw technical set-up of the environment and the learning outcomes. Process evaluation takes an in-depth look at the factors that facilitate or impede learning within the system. My study will need to focus on the process, with a possibility for pairing this up with outcome evaluations from colleagues.

4. Evaluation techniques
The authors seem to use the term objective in place of quantitative; they also refer to qualitative and mixed-method approaches. An important distinction between the two main forms is made – objective approaches deliberately remove context from the data, focusing on what could be directly comparable between other studies. Qualitative studies allowed for more direct insights for the learning processes within a community. I will be focusing on qualitative approaches, although there is potential for identifying the best factors to use in future studies for building up an objective measure of communities in future.

Conclusions

This gives a very good critical analysis of the factors at play in evaluating OLCs, and can serve as a guide point relating to the higher level discussions of Newby (2010), Cresswell (2009) and Colquhoon (2006). The authors also point out a good number of shortcomings in current research. Partly these are due to the constraints of researchers performing their studies for their own purposes, rather than to sit conveniently into the wider body of research. They also point out that the offline interactions between participants play a large role in the actual learning process, and these are very difficult to find any record of. There are also no studies that show how a community has evolved over time.

Long term goals of the researchers are to establish a framework for understanding OLCs, possibly towards a central theory. They make no reference to the five stage model identified by Salmon (2003); they do refer to phases of community development (Palloff and Pratt, 1999). It will be useful to pursue this systematic approach for research to inform my own understanding, and note if any similarities or contradictions with Salmon’s model emerge, which has previously been central to my understanding of online learning interactions.

Reference:

• Colquhoon, D. (2006). Research Methods in Education Contexts. University of Hull.
• Creswell, J. W. (2009). Research Design: qualitative, quantitative and mixed methods approaches. (3rd edition) Sage.
• Ke, F. and Hoadley, C. (2009). Evaluating Online Learning Communities. Educational Technology Research and Development, 57(4), pp.487-510.
• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.
• Salmon, G., 2003. E-Moderating: The Key to Teaching and Learning Online. 2nd ed. London: Routledge-Falmer.

Thoughts on Conrad

I've been going into more details on my literature review, and considering how other authors structure their writing, particularly from the perspective of what I find useful, and how it might influence my own writing. The first article is an interpretive study by Conrad (2002).

 

Conrad makes the point that while quantitative studies can give a useful overview of the area, understanding the experiences of users is a priority for development of communities. Although my context is somewhat different, her point about learners creating their own lines of defence sounds quite telling in the light of some initial comments observed in my study. Without understanding what barriers people put up, we can’t expect to engage meaningfully with them! Likewise, her point about our research agendas being shaped by our worldview matches with my research being guided by exploration of how network effects are re-shaping society, and the subsequent effects on education systems, both technological and organisational.

Writing a section about who, what, where, etc. is a useful step for grounding the paper, and setting the boundaries of the study, particularly the limitations of what might be achieved in the first place. She also breaks down the literature review itself to define different terms, building up how she wants the reader to understand her term of online community. She then examines each of the research questions in turn, looking at general patterns in responses, followed by particularly insightful comments by individuals. This approach will probably work very well for me, as I need to spot general patterns as well as bring out individual experiences.

Reference

Conrad, D. (2002). Deep in the Hearts of Learners: Insights into the Nature of Online Community. Journal of Distance Education, 17(1), 1-19

Reviewing the proposal

Considering points raised by Newby (2010, Ch. 2 & 14) to refine my research proposal has given me a mixture of outcomes. Some of these confirm that my basic methodology is correct, whilst some aspects might need tweaking.

Why this research?

My primary aim in this research is gaining a qualification – however I do need to relate the research to my work context, which has been a source of tension since in future I will need to contribute to projects that further divisional goals more directly.

Why am I interested in this area?

We are aiming to move towards a blended training model, with online and face-to-face components.  The use of forums to encourage interactions among participants has been running for some time, with uncertain results.

Why do I think that it is important?

The building of learning communities with increase responsibility for their own learning will become important to sustain learning across the organisation. Our current resourcing will not enable us to meet the continuously growing demand.

Will I be able to write enough to cover my task?

I have chosen this learning program because I know that there is the potential for enough research data to be gathered for a number of cohorts, and I can combine research on the participants with benefit for practitioners.

What experience do I have of this area?

I have made use of forums in my own study and introduced the idea to colleagues through an explorative example. This led to the introduction of forums in the context that I am researching, and I have been involved in an advisory and technical capacity since then.

Who else thinks this is worth exploring?

There is a slight difference of opinion amongst my colleagues as to the value of exploring this area further. Some of my colleagues however have expressed an interest in gaining insights into this area, and even those who don’t fully support it at least acknowledge their frustrations that the use of forums is problematic.

Creating the context for the proposal

Will it deliver on our personal goals?

Although it might meet the requirements of a modular assessment, it may not have a high enough ‘impact potential’ for further progression in some respects. The project should therefore be regarded as a stepping stone for future research efforts – my key learning points will be how successful the approach is in educating practitioners and giving further insights into a learning program. Once my methodology has been established I will then look for higher impact applications.

Does the topic have a political dimension?

Online education in general is an area of intense interest, along with the tools and media for achieving success. Political commentators and politicians may well have opinions on the area, but they are likely more interested in conventional learning matters at this time.

How much do we know about the area?

As stated previously, I have been involved with the use of online discussion for learning in a variety of contexts, both academic and work related. My initial literature review has uncovered some very useful information about how online communities can be investigated methodically, hence my approach to this project.

Identifying the research boundaries

Total amount of time available

In this case the time available is dictated by the time constraints of the assessment task, and my personal time for writing up my results.

What help can we expect from others?

The work does need to be primarily my own due to the assessment criteria, and the data collection should be fairly straightforward due to sending out questionnaires by e-mail. I do need to consider how much time it might take participants to answer my questionnaires, given that we will be approaching a time of higher workload.

Do we need specialist help?

No outside help will be necessary in this case since the data collection and interpretation is of my own devising, and I have sufficient access to journal articles at this time.

What if things don’t go according to plan?

I believe I have sufficient safety in terms of preserving data and the actual collection. My concern is with the possible response rate of participants within the time frame. For this reason I had already opted to research a program that has had multiple cohorts, and also the interactions have been digitally recorded due to the nature of the medium.

Assessing ethical issues

This has already been covered in the blog post on principles to some extent. As a summary, I will be carrying out my research within my workplace and sharing those results with colleagues. One potential issue could come from the sharing of participants’ responses to a wider audience, from a forum that was set up as a confidential area. For this reason I will need to ask colleagues about how to use their responses. I will treat responses anonymously by default, possibly with an option for their responses to not be quoted in reports. All postings on forums will remain confidential.

Preparing a case for a research programme

This research programme is for assessment purposes, but setting out my thinking will be of use in future, especially if I want to keep research as a positive influence on our evaluation strategy.

Getting an idea for research

From the sources of stimulus I have identified several different strands to my research idea & motivation:

1.   Literature

My review of the literature has revealed that there are a number of different theories cited by researchers, in some cases not including those that I have used as a theoretical basis for my own learning. Because of this I am wary of using quantitative methodology to tie results to a particular theory

2.   Policy

Ongoing interest in the field of online education interests me, particularly from the point of view of keeping practitioner education a key issue. Research that is carried out purely from a top-down institutional perspective is unlikely to produced sustainable benefits for the education system or society.

3.   Experience

Following on from previous points, my experience of online education is that it is most powerful as a constructivist environment, to support communities of inquiry and learning by doing.

Potential to finish research project

For this assessed research project I am confident that I can deliver the required results, and show that my methodology for design and data collection has been adequate. Questions about the impact of my research will need to be addressed for future efforts, since online learning projects will be focussed on areas of greatest business interest.

Arguing the case for the research

(a) Can the topic convince our judges?
The project should sufficiently convince my academic judges, although from a work perspective I will need to convince colleagues that the project has sufficient merit as a developmental tool for future work, and show that my approach to sharing results and insights can benefit the wider team.

(b) Is it feasible for me to undertake it?
I believe that this research project is well within my capabilities, and can be completed satisfactorily within the given time frame.

(i)  The field of enquiry
Research in this area is nothing particularly new, since there has been significant research into the field by academic researchers. Neither should this be considered as a theoretical study, although the context will always be different from others. The real interest here is how policies of using online education methods are succeeding against our objectives, and how the use of these approaches compares between academic origins and the actual implementation in the workplace.

(ii) The research issue / question
Related to the issues from previous questions, my key questions for research are based around the experiences of people participating in the use of new types of media, and ensuring that practitioner approaches can match the desire for business results.

Making sense of what we have

My interpretation of results may well draw on theory, although this will not be the primary lens. My main focus will be on judging whether activities have met objectives and how they can be modified for improvements, and also to identify what course of action should not be taken. My approach to truth here is realising how others perceive it, and using this to inform practice.

References

• Newby, Peter (2010).  Research Methods for Education Chapter 2 p.31-65