Re-affirming the project outline

My project outline has remained a little vague, so I thought I would try to add more details to it...

We currently provide secondary qualifications for 14 - 19 year olds. This work is dependent on a network of some 35,000 teachers and other experts to help set and mark examinations, with technology playing an increasingly pivotal role to ensure fast and scalable transfer of marks. We are continually exploring new technologies for mark capture and transfer to ensure the best possible service for candidates.

Trialling and adopting a new marking technology requires training provision for large numbers of examiners. Previous technology adoptions have initially been dependent on government funding for their initial success, but this funding is no longer available. Training provision has increasingly moved towards the creation of online software demonstration videos and interactive simulations, now hosted on a secure Learning Management System (LMS).

We are currently piloting a new marking technology with a very small number of examiners who have received face-to-face training, and are now looking to move to exclusively online training as soon as possible. With previous technology adoptions the online provision has been developed largely through internal discussion after face-to-face training and released without a live test for examiners. For this project, the online learning materials are being developed alongside the first live pilot of the technology, with an opportunity for early testing and feedback.

We will be using action research methods to inform improvements to the online learning materials and identify additional support methods prior to general release. This research will be expanded for the live use of the software during the summer examination series, with a view to providing both evaluation of success and action research for practitioner development.

Architecture revisited

I'm going to take another look at the Learning Architect approach for my project, but this time from the perspective of staff offering support to examiner. Now this covers a wide variety of job roles, so I'll try to cover as many angles as I can, and think of how we might make best use of the different areas. This post will probably come across as a little scattered at the time of writing - it's primarily intended for some reflection after the event!

From the top down:

Experiential
Staff will naturally be taking part in performance appraisals so it will be worth considering how their involvement with the online marking will be judged - relating this to our cornerstone behaviours. Making sure that good ideas are credited will help to embed good behaviours.

On-demand
Producing performance support materials that relate to specific roles - these could be as simple as one-sheet reference guides or checklists for particular tasks that can be printed out. These will include the materials that are provided for examiners so that staff can advise examiners who are under pressure, and could also be expanded to include common problems that are encountered by the team leading the initiative.

Non-formal
Staff will have access to the same rapid e-learning as examiners via the LMS, although perhaps there may be a case for adding in some modules that relate to specific roles? This could also be covered in mini-workshops for staff - though possibly this is a continuous blur into on-job training. The main focus should be on discussing particular problems, with the knowledge transfer aspect left to e-learning to be looked at before the sessions.
Other possibilities are webinars to cover discussion of emergent issues, although this is probably more likely to be done on an individual basis and difficult to capture.

Formal
Classroom courses will undoubtedly be offered in some form - although these might actually be better described as mini-workshops. The opportunity for collaboration is limited, and the focus is likely to be on knowledge transfer, which is better left to the e-learning modules. Whether these are regarded as formal self-study or rapid is a debate for elsewhere!
Our main nod to formal learning should be to ensure that business goals are made explicit, and that staff know how their role relates to achieving them. Assessment will be based on accomplishing objectives, and nothing more!

From the bottom up:

Experiential
Encouraging personal reflection or reflection with others within or across work teams will help to ensure that lessons are learned well. This will be dependent on engaging effectively with line managers.

On-demand
The use of forums and wikis is one area where we might be able to advance the information sharing between colleagues, although there would probably need to be some moderation of comments and content to ensure accuracy, and many colleagues would probably prefer to stick to more formal channels that they are used to monitoring. The use of a wiki may be worth pursuing for future, but effective guidance would have to be in place, and this would have to take a back seat until other top-down measures are in place.

Non-formal
Nothing here for now

Formal

Nothing here for now

Overall, I believe the need for bottom-up learning is less immediate in this context - the tasks and goals are generally quite fixed, rather than fluid. Our key focus for bottom-up learning should centre around making tacit knowledge transfer more readily to explicit knowledge. This will stem primarily from the staff leading the project, but careful consideration of affected parties will help us know where best to direct our efforts.

The 'Carry On' factor

Image: freeimages

Following on from my thoughts about quantitative research, I'm looking at some of the dependent variables that will come into play, and thinking about how I might go about analysing them.

Intention to continue examining & job satisfaction

This is an extremely important factor for exam boards, as they are dependent on a large network of examiners to make our examinations possible. Meadows (2004) identified four factors that affect examiners' attitudes towards their jobs:

1. The pressure and stress of examining
2. Insight gained from examining
3. Support from awarding body and senior examiners
4. Pay

However, Meadows found that only the pressure and stress of examining, and the level of support received, predicted intention to continue examining; however pay did affect examiners' job satisfaction. One of the key sources of stress came from balancing examining duties with regular work, with the report recommending that resources should be diverted to lobbying for examiners to be given more time away from teaching to examine, in order to improve retention. Improving the level of support was also a recommendation, although the report notes that this would be less cost effective, since most examiners were already relatively satisfied with the support they received. Increasing pay would improve job satisfaction, but the report states that this would not improve retention.

The introduction of software tools

Tremain (2011) followed up this work to consider how the situation had changed after the introduction of electronic marking and online standardisation. The study looked at the factors that influence the satisfaction that examiners express about their work, and highlighted three factors underpinning examiners' intention to continue:

1. The relationship between examining work and work outside examining
2. The pressures of examining and support received
3. The incentives for examining

The study states that although there is no imminent threat to examiner retention, future threats include the increasing use of online tools, which can contribute to examiners feeling unsupported or undervalued. Job satisfaction is considered to be more important in retention than reward for the majority of jobs, with social interaction and appropriate challenge being considered particularly valuable. The adoption of online tools had contributed to a sense of isolation amongst examiners, and also made the work more routine - although the reliability of marking has actually increased as a result.

A further study (Tremain, 2012) also set out to evaluate how specific factors involved in online marking & standardisation contributed to examiner satisfaction. This concluded that there was no significant difference in intention to continue marking between examiners who were standardised using face-to-face or online methods. Examiners who had marked using a mixture of paper and online methods showed a very slight increase in intention to continue examining. However, it was noted that the results were confounded by the different subjects and levels of experience amongst the participants.

Variables that we may be able to influence, and how:

• Support received. By considering the different levels of support that are currently offered from the contextual model for learning (Shepherd, 2011) and identifying possible gaps, we may be able to improve the support offering for examiners in a rational way. I have already laid out some initial thoughts for this approach.
• Insight gained from examining. Making key insights from senior examiners available in a digital form which can be shared more easily online, for instance through learning management systems or webinars, could help to ensure efficient dissemination of relevant information.
• Social interaction. This is a long term goal that our organisation may want to consider for retaining examiners. Although we are increasingly unable to provide opportunities for examiners to meet in a face-to-face setting, there are possibilities for facilitating some more informal interaction around scheduled events. One of my colleagues is keen to run webinars for examiners to gain insight from senior examiners, and careful use of online chat could help to provide a better sense of community.

Any or all of these methods could be attempted, with measurement of the effect on intention to continue, and also examiner performance, being undertaken to determine effectiveness. One concern I have is that apparent failure to make a difference at first might result in a loss of enthusiasm for innovation, hence there would need to be trust established with stakeholders for future improvements. Undertaking action research alongside quantitative measurements to demonstrate a rational approach would be key to successful establishment of such trust.

References:

• Meadows, M., 2004. Measures of examiner job satisfaction as predictors of intention to continue examining. Available at: https://cerp.aqa.org.uk/research-library/measures-examiner-job-satisfaction-predictors-intention-continue-examining [Accessed December 8, 2013].
• Shepherd, C., 2011. The new learning architect, Chesterfield, U.K.: Onlignment.
• Tremain, K., 2011. Carry on examining: what predicts examiners’ intentions to continue examining? Available at: https://cerp.aqa.org.uk/research-library/carry-examining-what-predicts-examiners-intentions-continue-examining [Accessed December 7, 2013].
• Tremain, K., 2012. Carry on examining: further investigation. Available at: https://cerp.aqa.org.uk/research-library/carry-examining-further-investigation [Accessed December 7, 2013].