Five questions

Image: freeimages

I'm currently working through some questions for framing qualitative research (Mason, 2002), which I hope will help me to better frame my research.

The social 'reality': Your Ontological Perspective

What is the nature of the phenomena, or entities, or 'social reality' that I wish to investigate?

Elements from Table 1.1 that appeal:

• People, social actors, motivations, identities, cultural or social constructions
• Experiences, development, behaviours, interactions, social processes
• Institutions, markets, societies, organisation, connectedness, multiple realities or versions + tribes, networks

These are more relevant to my overall world-view, so I will generate an additional set specific to the current project:

• People, understandings, perceptions,attitudes, thought
• Experiences, development, actions, interactions, situations, rules
• The 'material', groups, organisation

Knowledge and Evidence: Your Epistemological Position

What might represent knowledge or evidence of the entities or social 'reality' that I wish to investigate?

Experiences of users will allow me to link my design approach to how people perceive the materials and support framework - affects my ability to rationally design or re-design materials and approaches in response to user feedback.

The research would benefit from some objective measures of performance to show that knowledge about systems and procedures has successfully transferred, rather than asking for user ratings - allows for genuine accountability in our training approach.

Your Broad Research Area

What topic, or broad substantive area, is the research concerned with?

This research topic is concerned with understanding what experiences examiners have when using online resources and support to adopt new software for marking and standardisation, and any different procedures that must be adopted. My focus is on interpreting results in such a way as to allow continuous improvement of the design of such materials, and to understand how much effect the strategy has on learning, as evidenced through performance. I do not have control over the precise methods for measuring performance, but am able to access such data.

Your Intellectual Puzzle and Your Research Questions

What is the intellectual puzzle?

How do I show that my learning intervention has had the desired impact, and how can I rationally design better approaches in future, or re-design to off-set any shortcomings?

What do I wish to explain or explore?

I wish to explain how the support approaches link to performance and attitudes among the examiners, and explore how to improve both in the future.

What type of puzzle is it?

This is a causal / predictive puzzle.

Your Research Questions

What are my research questions?

For gaining insights from examiners taking part in the live pilot I'm thinking along the lines of:

• Describe your experiences of using the software to carry out standardisation of marking
• Describe your experiences of using the software to record and submit your marks
• Describe your experiences of communicating with your senior examiner during the marking period

For improving on my online materials:

• Describe your experience of using the online resources to prepare you for marking
• Describe your experience of using the online resources and printable materials during the marking period

More of these to come - from previous experience I find that it's best to mull over these and look at what I've written down with a fresh perspective later.

Your Aims and Purpose

What is the purpose of my research? What am I doing it for?

The purpose of this research is to ensure the timely and accurate delivery of high quality marking for candidates sitting national examinations. The research will be done on behalf of my employer, to benefit examiners whilst delivering the marking and help to maintain motivation through consideration of the support offered. By achieving these objectives it is also expected that examiner performance will improve, which will be judged through the assessments from senior examiners and staff. The timeliness and accuracy of results could also be interrogated relative to expected deadlines and the number of examiners stopped or re-marked.

References

• Mason, J., 2002. Qualitative researching, London: SAGE Publications.

Architecture revisited

I'm going to take another look at the Learning Architect approach for my project, but this time from the perspective of staff offering support to examiner. Now this covers a wide variety of job roles, so I'll try to cover as many angles as I can, and think of how we might make best use of the different areas. This post will probably come across as a little scattered at the time of writing - it's primarily intended for some reflection after the event!

From the top down:

Experiential
Staff will naturally be taking part in performance appraisals so it will be worth considering how their involvement with the online marking will be judged - relating this to our cornerstone behaviours. Making sure that good ideas are credited will help to embed good behaviours.

On-demand
Producing performance support materials that relate to specific roles - these could be as simple as one-sheet reference guides or checklists for particular tasks that can be printed out. These will include the materials that are provided for examiners so that staff can advise examiners who are under pressure, and could also be expanded to include common problems that are encountered by the team leading the initiative.

Non-formal
Staff will have access to the same rapid e-learning as examiners via the LMS, although perhaps there may be a case for adding in some modules that relate to specific roles? This could also be covered in mini-workshops for staff - though possibly this is a continuous blur into on-job training. The main focus should be on discussing particular problems, with the knowledge transfer aspect left to e-learning to be looked at before the sessions.
Other possibilities are webinars to cover discussion of emergent issues, although this is probably more likely to be done on an individual basis and difficult to capture.

Formal
Classroom courses will undoubtedly be offered in some form - although these might actually be better described as mini-workshops. The opportunity for collaboration is limited, and the focus is likely to be on knowledge transfer, which is better left to the e-learning modules. Whether these are regarded as formal self-study or rapid is a debate for elsewhere!
Our main nod to formal learning should be to ensure that business goals are made explicit, and that staff know how their role relates to achieving them. Assessment will be based on accomplishing objectives, and nothing more!

From the bottom up:

Experiential
Encouraging personal reflection or reflection with others within or across work teams will help to ensure that lessons are learned well. This will be dependent on engaging effectively with line managers.

On-demand
The use of forums and wikis is one area where we might be able to advance the information sharing between colleagues, although there would probably need to be some moderation of comments and content to ensure accuracy, and many colleagues would probably prefer to stick to more formal channels that they are used to monitoring. The use of a wiki may be worth pursuing for future, but effective guidance would have to be in place, and this would have to take a back seat until other top-down measures are in place.

Non-formal
Nothing here for now

Formal

Nothing here for now

Overall, I believe the need for bottom-up learning is less immediate in this context - the tasks and goals are generally quite fixed, rather than fluid. Our key focus for bottom-up learning should centre around making tacit knowledge transfer more readily to explicit knowledge. This will stem primarily from the staff leading the project, but careful consideration of affected parties will help us know where best to direct our efforts.

Re-examining my worldview

Image: ariss via freeimages

Research Philosophy

How do I identify with four of the main worldview areas, as discussed by Newby?

Scientism and Positivism

This school of thought demands predictable cause and effect - not easy to establish where people are concerned. Once people realise that others are able to predict their behaviour in any way, they tend to change their behaviour to avoid manipulation! It may be beneficial to look for patterns if you have a critical interest, such as a business issue, although this may actually overlap with more humanistic traditions (see below) because you will be seeking a model that is 'fit for purpose' rather than trying to establish an objective truth.

This is the case for the research carried out by Meadows (2004) and Tremain (2011, 2012), where the critical factors affecting examiner retention and job satisfaction needed to be identified. Establishing an all-encompassing theory or truth (positivism) may be too ambitious - and would be clouded by personal interests - identifying and monitoring the relevant factors was achieved through careful selection of questions and interpretation of data.

Future reading: Discussion of the work of Karl Popper (Newby, Ch.3)

Humanism, Phenomenology and Existentialism

Whist positivism demands an objective external truth independent of human influence, humanism treats 'truth' as a social construction, where one culture's truth may not be another's. This ties well with my understanding of social evolution (Hobson, 2012; Ronfeldt, 1996, 2012a,b) whereby societies develop distinct cultures of varying complexity through the addition of different forms of organisation. Conflicts arise between or even within societies due to disagreements about how society should be ordered (or not) by hierarchical institutions and free market policies. Ronfeldt (2013) acknowledges that even the term 'tribe' in his T-I-M-N framework is frequently contested by others, and progresses the model by engaging with and incorporating these disagreements into his model.

Phenomenology focuses on individual and collective experiences to form a rational basis for future action, by probing the differences between 'perceived' and 'experienced' world. Methods include description observation, reporting and reflection.

Existentialism centres around seeking to understand the world from a personal perspective, driven by conviction and desire. Methods could extend to asking participants for other forms of 'data' such as pictures, videos and stories of their lives - anything that conveys their viewpoint in richer detail than a questionnaire could. Existentialism has been applied to curriculum design, by designing a curriculum that centres around self-discovery.

Critical theory

This theory is concerned with political beliefs, particularly those that are left-of-centre and seeking to change society by making people aware of their circumstances in order to liberate them. Research carried out within a critical theory framework seeks to observe and expose individual facts which can then be combined to form an argument for change.

Aside: I am particularly interested in the references to Karl Marx, as this led to a model for social evolution around the struggle for control of wealth - somewhat akin to Ronfeldt's description of the transition between biform (T+I) and triform (T+I+M) societies.

Applying critical theory to education has been used by academics to attack the present model for education as being primarily based around servicing the needs of the capitalist economy, and thus serves only to reinforce the inequalities that it inevitably generates. However positivists would undoubtedly attack this approach because it proposes a hypothesis and seeks to prove it.

Postmodernism

This approach distinguishes itself from the 'modernist' approaches by rejecting the 'modernist' assumption that there is a single explanation for things, which leads to a natural order. For education, this almost rejects the need for theories of education, since such theories are rooted in modernist approaches! This approach does overlap with my TIMN world-view, because it is by nature multi-layered, and acknowledges that 'people and organisations can play several and sometimes conflicting roles', in keeping with the different types of organisations and the interfaces between them.

Postmodernism bears a great deal of similarity to the network (+N) principles described by Ronfeldt, with researchers seeking to act more as nodes in a network, unpicking their assumptions and sharing data that reflects their local situation, with the ideal that the value of the network increases not simply with the number of nodes but the number of connections between them.

Future reading: Steven Johnson: Future Perfect 

Summary

My adoption to the T-I-M-N framework in my outlook leads me to cross several of the boundaries of research philosophy, but perhaps I identify more strongly than I realised with the post-modernist approach for rejecting the established order with its demanded polarisation of political outlooks.

References:

• Hobson, J. (2012). The problems with educating a quadriform society. Thinking and learning in the digital age [blog] 7 August. Available at: <http://jimmyhob.blogspot.co.uk/2012/08/the-problems-with-educating-quadriform.html>
• Meadows, M., 2004. Measures of examiner job satisfaction as predictors of intention to continue examining. Available at: https://cerp.aqa.org.uk/research-library/measures-examiner-job-satisfaction-predictors-intention-continue-examining [Accessed December 8, 2013].
• Ronfeldt, D. (1996). Tribes, Institutions, Markets, Networks – A Framework about Societal Evolution. Available at: <http://www.rand.org/pubs/papers/P7967.html>[Accessed September 2012]
• Ronfeldt, D. (2012a). TIMN in 20 minutes: a video presentation. Visions from two theories [blog] 23 May. Available at: <http://twotheories.blogspot.co.uk/2012_05_01_archive.html> [Accessed March 2013]
• Ronfeldt, D. (2012b). Q’s & A’s about “TIMN in 20 minutes” (4th of 7): creation of a new sector? Visions from two theories [blog] 29 August. Available at: <http://twotheories.blogspot.co.uk/2012/10/qs-as-about-timn-in-20-minutes-4th-of-7_29.html> [Accessed March 2013]
• Ronfeldt, D. (2013). Further points about “tribes” (T) — plus a new proposition about TIMN as a whole. Visions from two theories [blog] 10 April. Available at: <http://twotheories.blogspot.co.uk/2013/04/further-points-about-tribes-t-plus-new.html> [Accessed March 2013]
• Tremain, K., 2011. Carry on examining: what predicts examiners’ intentions to continue examining? Available at: https://cerp.aqa.org.uk/research-library/carry-examining-what-predicts-examiners-intentions-continue-examining [Accessed December 7, 2013].
• Tremain, K., 2012. Carry on examining: further investigation. Available at: https://cerp.aqa.org.uk/research-library/carry-examining-further-investigation [Accessed December 7, 2013].

The 'Carry On' factor

Image: freeimages

Following on from my thoughts about quantitative research, I'm looking at some of the dependent variables that will come into play, and thinking about how I might go about analysing them.

Intention to continue examining & job satisfaction

This is an extremely important factor for exam boards, as they are dependent on a large network of examiners to make our examinations possible. Meadows (2004) identified four factors that affect examiners' attitudes towards their jobs:

1. The pressure and stress of examining
2. Insight gained from examining
3. Support from awarding body and senior examiners
4. Pay

However, Meadows found that only the pressure and stress of examining, and the level of support received, predicted intention to continue examining; however pay did affect examiners' job satisfaction. One of the key sources of stress came from balancing examining duties with regular work, with the report recommending that resources should be diverted to lobbying for examiners to be given more time away from teaching to examine, in order to improve retention. Improving the level of support was also a recommendation, although the report notes that this would be less cost effective, since most examiners were already relatively satisfied with the support they received. Increasing pay would improve job satisfaction, but the report states that this would not improve retention.

The introduction of software tools

Tremain (2011) followed up this work to consider how the situation had changed after the introduction of electronic marking and online standardisation. The study looked at the factors that influence the satisfaction that examiners express about their work, and highlighted three factors underpinning examiners' intention to continue:

1. The relationship between examining work and work outside examining
2. The pressures of examining and support received
3. The incentives for examining

The study states that although there is no imminent threat to examiner retention, future threats include the increasing use of online tools, which can contribute to examiners feeling unsupported or undervalued. Job satisfaction is considered to be more important in retention than reward for the majority of jobs, with social interaction and appropriate challenge being considered particularly valuable. The adoption of online tools had contributed to a sense of isolation amongst examiners, and also made the work more routine - although the reliability of marking has actually increased as a result.

A further study (Tremain, 2012) also set out to evaluate how specific factors involved in online marking & standardisation contributed to examiner satisfaction. This concluded that there was no significant difference in intention to continue marking between examiners who were standardised using face-to-face or online methods. Examiners who had marked using a mixture of paper and online methods showed a very slight increase in intention to continue examining. However, it was noted that the results were confounded by the different subjects and levels of experience amongst the participants.

Variables that we may be able to influence, and how:

• Support received. By considering the different levels of support that are currently offered from the contextual model for learning (Shepherd, 2011) and identifying possible gaps, we may be able to improve the support offering for examiners in a rational way. I have already laid out some initial thoughts for this approach.
• Insight gained from examining. Making key insights from senior examiners available in a digital form which can be shared more easily online, for instance through learning management systems or webinars, could help to ensure efficient dissemination of relevant information.
• Social interaction. This is a long term goal that our organisation may want to consider for retaining examiners. Although we are increasingly unable to provide opportunities for examiners to meet in a face-to-face setting, there are possibilities for facilitating some more informal interaction around scheduled events. One of my colleagues is keen to run webinars for examiners to gain insight from senior examiners, and careful use of online chat could help to provide a better sense of community.

Any or all of these methods could be attempted, with measurement of the effect on intention to continue, and also examiner performance, being undertaken to determine effectiveness. One concern I have is that apparent failure to make a difference at first might result in a loss of enthusiasm for innovation, hence there would need to be trust established with stakeholders for future improvements. Undertaking action research alongside quantitative measurements to demonstrate a rational approach would be key to successful establishment of such trust.

References:

• Meadows, M., 2004. Measures of examiner job satisfaction as predictors of intention to continue examining. Available at: https://cerp.aqa.org.uk/research-library/measures-examiner-job-satisfaction-predictors-intention-continue-examining [Accessed December 8, 2013].
• Shepherd, C., 2011. The new learning architect, Chesterfield, U.K.: Onlignment.
• Tremain, K., 2011. Carry on examining: what predicts examiners’ intentions to continue examining? Available at: https://cerp.aqa.org.uk/research-library/carry-examining-what-predicts-examiners-intentions-continue-examining [Accessed December 7, 2013].
• Tremain, K., 2012. Carry on examining: further investigation. Available at: https://cerp.aqa.org.uk/research-library/carry-examining-further-investigation [Accessed December 7, 2013].

Architecture

Going off on a little bit of a tangent, it's time to take a look at what support is being offered to our examiners through the e-learning provision, and how it could be improved. Shepherd (2011) provides a contextual model for learning, based on four contexts: experiential, on-demand, non-formal, and formal; and two perspectives: top-down and bottom-up.

• Experiential learning is learning from as opposed to learning to. We have to be actively engaged with our task and - one hopes - reflecting on our successes and failures. There are a great wealth of lessons that can be learned while examining - Meadows (2004) and Tremain (2011) report that examiners consistently cite the insight gained from examining as one of the key benefits. Considering how to support or encourage such reflections might help to improve examiner performance.
• On-demand learning is learning to perform a particular task and acquire the necessary knowledge, at the point of need or 'just-in-time'. Depending on how far in advance our examiners access the learning materials, they could be regarded in this fashion, although they are probably better used in the following category.
• Non-formal learning is also learning to but with a more relaxed time frame, where employers take steps for employees to be prepared in the medium to long term, and is sometimes labelled as 'just-in-case' learning to set it apart from on-demand. It is distinguished from formal learning below by not being packaged as a formal 'course', which will be the case for this intervention, although there may be something to be said for carefully considering how the materials are to be presented to examiners.
• Formal learning is defined by clearly set learning objectives, a means of assessment, and usually some kind of qualification. We definitely don't offer a qualification for learning to examine (perhaps some would say we should?!), assessment would be somewhat laborious unless it were to be done covertly through completion, and the learning objectives are difficult to define. It is far easier here to think in terms of business objectives - 'There are no learning metrics, only business metrics' (Cross) That being said, it will be worth considering what examiners will expect to see and make sure that objectives are clearly stated.
• Top-down learning is aligned with employers' objectives, and is intended to ensure performance is not left to chance, and sets out to ensure that the requisite skills and knowledge can be acquired to do so. Our organisation is responsible for ensuring that results are delivered on time and accurately, with severe penalties possible for failing to do so.
• Bottom-up learning occurs because of employees' motivation to perform effectively. In addition to the motives around improved insight, examiners are drawn to the extra pay for examining, and improved promotion prospects in their teaching roles (Meadows, 2004).

Moving into specifics from Shepherd's model, there are several components that are either currently present in our business model, introduced explicitly through the e-learning provision, and some that perhaps should be there. The existing provisions focus exclusively on the top-down perspective:

• There are already performance appraisals built into our way of working (experiential);
• A help-desk is provided through our Contact Centre (on-demand);
• Examiners typically receive a certain amount of on-job training through contact with their supervisors (non-formal);
• For the live pilot, examiners will be receiving classroom courses (formal);
• The majority of users will have access to the rapid e-learning materials (non-formal). Note that I avoid referring to our e-learning as self-study e-learning (a formal intervention), which by definition should provide 'instant and individualised feedback', something that is far beyond the scope of our planning.

If we open ourselves up to the full range of possibilities from Shepherd's model, I would be tempted to add the following methods:

• Webinars could be used to convey a lot of the material and briefing that might take place in a face-to-face context, without examiners having to travel to a central location, and allow for some questions & answers
• Online video could be a powerful tool for engaging examiners with the task at hand, especially if delivered by senior examiners involved in the pilot. The message would have to be particularly clear, relevant and to-the-point, requiring serious consideration before asking for this intervention.
• Performance support materials could be leveraged for contact centre staff and senior examiners, drawing on key lessons from the live pilot.
• Forums could be used amongst contact centre staff to post common questions from examiners; there could also be forums available to examiners to make common or emergent solutions available.
• Of course I would love it if someone other than myself found a reason to keep a blog ...

That's all for now I think, I'll look at specific applications in future posts....

References:

• Meadows, M., 2004. Measures of examiner job satisfaction as predictors of intention to continue examining. Available at: https://cerp.aqa.org.uk/research-library/measures-examiner-job-satisfaction-predictors-intention-continue-examining [Accessed December 8, 2013].
• Shepherd, C., 2011. The new learning architect, Chesterfield, U.K.: Onlignment.
• Tremain, K., 2011. Carry on examining: what predicts examiners’ intentions to continue examining? Available at: https://cerp.aqa.org.uk/research-library/carry-examining-what-predicts-examiners-intentions-continue-examining [Accessed December 7, 2013].

Time for a numbers game?

Image: freeimages

Following on from my last blog post, I'm re-treading the sequence of reading from our Research Methods module to get my bearings again, and I'm coming back to the question of qualitative vs quantitative research. While I strongly identified with the Action Research methodology on my last project, it's worth deliberately opening up my mind to new possibilities, especially as there will be strong interest in some kind of numerical data from colleagues and external auditors if we are questioned on our approach.

So before I start to choose which quantitative disciplines I might wish to draw on, I'll look at the key aspects of quantitative research, consider those that appeal to me, and those I wish to avoid.


Concern with theory

Relating my findings to theory will be helpful to ensure some kind of tethers to related work, but there's a danger of getting obsessed with reproducibility and control here. When you're moving into the realm of on-demand learning, you can't guarantee learning outcomes, nor indeed that learners will even access the materials or activities that you produce for them. Newby (2010, p.96) acknowledge the limitations for educational researchers trying to identify pattern and control influences, as they are only able to view a small part of the overall education system. I would prefer to think in terms of Praxis (Wheeler, 2013), which requires practitioners to consider how closely their practice overlaps with the theories they identify with.

Concern with proof

Here lies one of the real problems for educational research - although I understand that establishing proof would give greater peace of mind, the complexity and ambiguity of the situation makes this extremely difficult:

• The situation I face will not be the same as another practitioner does, even if our verbal descriptions of it seem similar to the untrained eye
• The next situation that I (and the learners) face will not be the same as this one, even if it's 'just another e-marking system'
• The time needed to establish proof would be completely at odds with the time pressures for the project, where the learning is 'on-demand'.

The best that I can hope for is to show that using theories to guide my design leads to a dependable business outcome, and that particular methods or techniques are better suited to my situation.

Identification of variables

This is one of the key aspects of quantitative theory that I see as helpful. Although my control over most independent variables involved will be limited to say the least, it will definitely be helpful to at least make some systematic efforts to identify variables in the design of materials that may be having an effect, and to measure any dependent variables which are of interest. Our particular concerns would be the performance of examiners, and intention to continue based on their experiences. Attempting to correlate these with participation in the different aspects of the support might yield useful insights into which components have succeeded, but this would have to be linked to effective practice in design.

Simply saying that an approach should be abandoned because it doesn't seem to have an effect in this situation would be potentially misleading without some understanding as to why. Creswell (2009, p.49) refers to confounding variables (e.g. discriminatory attitudes) that can come into play, which I have had some experience of when trying to introduce online learning methods in the past. Participants who are negative about the use of the tools go to great lengths to discredit them when given the opportunity to do so, whilst the majority of participants actually acknowledge a positive effect.

Conclusion

This project will benefit from the use of some quantitative approaches to analysing data about examiner performance and intention to continue, but these will need to be paired effectively with qualitative methods to understand what dependent variables relating to the choice and design of approaches might be influencing the outcomes. My next blog post will focus on the type(s) of quantitative research methods might be useful, followed by a look at rational design approaches for the learning provision.

References:

• Creswell, J. W. (2009). Research Design: qualitative, quantitative and mixed methods approaches. (3rd edition) Sage.
• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.
• Wheeler, S. (2013). Praxis makes perfect. Learning with e's [blog] 31 October. Available at: <http://steve-wheeler.blogspot.co.uk/2013/10/praxis-makes-perfect.html>

Back to where it all began

Image: freeimages

After a year or so of working on staff development programmes and doing a lot of creative and novel work, I'm back to where my career in online learning really began: making tutorial videos and briefings for examiners. As always, the time pressure is intense and I'm largely working solo - while some might see this as stressful I'm actually looking forward to it, because it's a welcome chance to really reflect on the circumstances where I first honed my skill set.

'Experience: that most brutal of teachers. But you learn, my God do you learn.'

C. S. Lewis

The tools changed several times in the first few years. The first recording software we started using because we had already invested in related software - I won't specify which software here, because it may have moved on since. You could record your screen with live narration, it was possible to easily publish the material using a hyperlink, the results could be reasonable, but it wasn't without its flaws. Particularly the editing could be a real nightmare if you made any mistakes in your recording, and quality seemed to degrade with each edit.

The second time around we started using Techsmith Camtasia Studio, which was a massive revelation. Screen recording was a great deal sharper, the editing process was vastly improved and I quickly found the value of highlighting, captions, and zooming & panning the view to draw attention to relevant areas. We now had the freedom to publish to good quality video formats, and with backup from the web team we could publish videos to an orphan page for examiners to view.

Finally we moved on to Adobe Captivate, which we have stuck with ever since for software demonstrations. It's a lot more technical than other software, which may put some people off, but it's allowed me to move forward with creating more interactive material (particularly simulations). As we finally moved over to our own LMS, the software had what we needed to publish with all the e-learning information for SCORM packages.

I've learned to stay mindful of the advice from Henderson (2012) to avoid being trapped by the tools, and that of Toth (2012) to choose the right tool for the job, so I always look for opportunities to use different e-learning tools. However I am finding it harder to take on new tools as my time gets increasingly bound up in development, so perhaps now isn't the time to take on something new for the recording. Instead I'll be looking to draw on the advice from Shepherd (2011) around considering carefully the context of your learners and what support they might need. In subsequent posts I'll be drawing up outlines for the additional approaches that might be used, and the opportunities to draw in different tools.

I'm automatically thinking of using Action Research methodology, since it was successful for last project, but this talk of not being trapped by your tools has made me pause. Perhaps it's worth re-treading some of the exercises from the Research Methods course and make sure that Action Research, and indeed qualitative research, is the correct approach.

References:

• Henderson, A., 2012. Don't get trapped by your e-learning tools. In: Allen, M.W., 2012. Michael Allen’s E-learning annual 2012, San Francisco, Calif.: Pfeiffer.
• Toth, T.A., 2012. The right e-learning tool for the job. In: Allen, M.W., 2012. Michael Allen’s E-learning annual 2012, San Francisco, Calif.: Pfeiffer.
• Shepherd, C., 2011. The new learning architect, Chesterfield, U.K.: Onlignment.