Re-examining my worldview

Image: ariss via freeimages

Research Philosophy

How do I identify with four of the main worldview areas, as discussed by Newby?

Scientism and Positivism

This school of thought demands predictable cause and effect - not easy to establish where people are concerned. Once people realise that others are able to predict their behaviour in any way, they tend to change their behaviour to avoid manipulation! It may be beneficial to look for patterns if you have a critical interest, such as a business issue, although this may actually overlap with more humanistic traditions (see below) because you will be seeking a model that is 'fit for purpose' rather than trying to establish an objective truth.

This is the case for the research carried out by Meadows (2004) and Tremain (2011, 2012), where the critical factors affecting examiner retention and job satisfaction needed to be identified. Establishing an all-encompassing theory or truth (positivism) may be too ambitious - and would be clouded by personal interests - identifying and monitoring the relevant factors was achieved through careful selection of questions and interpretation of data.

Future reading: Discussion of the work of Karl Popper (Newby, Ch.3)

Humanism, Phenomenology and Existentialism

Whist positivism demands an objective external truth independent of human influence, humanism treats 'truth' as a social construction, where one culture's truth may not be another's. This ties well with my understanding of social evolution (Hobson, 2012; Ronfeldt, 1996, 2012a,b) whereby societies develop distinct cultures of varying complexity through the addition of different forms of organisation. Conflicts arise between or even within societies due to disagreements about how society should be ordered (or not) by hierarchical institutions and free market policies. Ronfeldt (2013) acknowledges that even the term 'tribe' in his T-I-M-N framework is frequently contested by others, and progresses the model by engaging with and incorporating these disagreements into his model.

Phenomenology focuses on individual and collective experiences to form a rational basis for future action, by probing the differences between 'perceived' and 'experienced' world. Methods include description observation, reporting and reflection.

Existentialism centres around seeking to understand the world from a personal perspective, driven by conviction and desire. Methods could extend to asking participants for other forms of 'data' such as pictures, videos and stories of their lives - anything that conveys their viewpoint in richer detail than a questionnaire could. Existentialism has been applied to curriculum design, by designing a curriculum that centres around self-discovery.

Critical theory

This theory is concerned with political beliefs, particularly those that are left-of-centre and seeking to change society by making people aware of their circumstances in order to liberate them. Research carried out within a critical theory framework seeks to observe and expose individual facts which can then be combined to form an argument for change.

Aside: I am particularly interested in the references to Karl Marx, as this led to a model for social evolution around the struggle for control of wealth - somewhat akin to Ronfeldt's description of the transition between biform (T+I) and triform (T+I+M) societies.

Applying critical theory to education has been used by academics to attack the present model for education as being primarily based around servicing the needs of the capitalist economy, and thus serves only to reinforce the inequalities that it inevitably generates. However positivists would undoubtedly attack this approach because it proposes a hypothesis and seeks to prove it.

Postmodernism

This approach distinguishes itself from the 'modernist' approaches by rejecting the 'modernist' assumption that there is a single explanation for things, which leads to a natural order. For education, this almost rejects the need for theories of education, since such theories are rooted in modernist approaches! This approach does overlap with my TIMN world-view, because it is by nature multi-layered, and acknowledges that 'people and organisations can play several and sometimes conflicting roles', in keeping with the different types of organisations and the interfaces between them.

Postmodernism bears a great deal of similarity to the network (+N) principles described by Ronfeldt, with researchers seeking to act more as nodes in a network, unpicking their assumptions and sharing data that reflects their local situation, with the ideal that the value of the network increases not simply with the number of nodes but the number of connections between them.

Future reading: Steven Johnson: Future Perfect 

Summary

My adoption to the T-I-M-N framework in my outlook leads me to cross several of the boundaries of research philosophy, but perhaps I identify more strongly than I realised with the post-modernist approach for rejecting the established order with its demanded polarisation of political outlooks.

References:

• Hobson, J. (2012). The problems with educating a quadriform society. Thinking and learning in the digital age [blog] 7 August. Available at: <http://jimmyhob.blogspot.co.uk/2012/08/the-problems-with-educating-quadriform.html>
• Meadows, M., 2004. Measures of examiner job satisfaction as predictors of intention to continue examining. Available at: https://cerp.aqa.org.uk/research-library/measures-examiner-job-satisfaction-predictors-intention-continue-examining [Accessed December 8, 2013].
• Ronfeldt, D. (1996). Tribes, Institutions, Markets, Networks – A Framework about Societal Evolution. Available at: <http://www.rand.org/pubs/papers/P7967.html>[Accessed September 2012]
• Ronfeldt, D. (2012a). TIMN in 20 minutes: a video presentation. Visions from two theories [blog] 23 May. Available at: <http://twotheories.blogspot.co.uk/2012_05_01_archive.html> [Accessed March 2013]
• Ronfeldt, D. (2012b). Q’s & A’s about “TIMN in 20 minutes” (4th of 7): creation of a new sector? Visions from two theories [blog] 29 August. Available at: <http://twotheories.blogspot.co.uk/2012/10/qs-as-about-timn-in-20-minutes-4th-of-7_29.html> [Accessed March 2013]
• Ronfeldt, D. (2013). Further points about “tribes” (T) — plus a new proposition about TIMN as a whole. Visions from two theories [blog] 10 April. Available at: <http://twotheories.blogspot.co.uk/2013/04/further-points-about-tribes-t-plus-new.html> [Accessed March 2013]
• Tremain, K., 2011. Carry on examining: what predicts examiners’ intentions to continue examining? Available at: https://cerp.aqa.org.uk/research-library/carry-examining-what-predicts-examiners-intentions-continue-examining [Accessed December 7, 2013].
• Tremain, K., 2012. Carry on examining: further investigation. Available at: https://cerp.aqa.org.uk/research-library/carry-examining-further-investigation [Accessed December 7, 2013].

The 'Carry On' factor

Image: freeimages

Following on from my thoughts about quantitative research, I'm looking at some of the dependent variables that will come into play, and thinking about how I might go about analysing them.

Intention to continue examining & job satisfaction

This is an extremely important factor for exam boards, as they are dependent on a large network of examiners to make our examinations possible. Meadows (2004) identified four factors that affect examiners' attitudes towards their jobs:

1. The pressure and stress of examining
2. Insight gained from examining
3. Support from awarding body and senior examiners
4. Pay

However, Meadows found that only the pressure and stress of examining, and the level of support received, predicted intention to continue examining; however pay did affect examiners' job satisfaction. One of the key sources of stress came from balancing examining duties with regular work, with the report recommending that resources should be diverted to lobbying for examiners to be given more time away from teaching to examine, in order to improve retention. Improving the level of support was also a recommendation, although the report notes that this would be less cost effective, since most examiners were already relatively satisfied with the support they received. Increasing pay would improve job satisfaction, but the report states that this would not improve retention.

The introduction of software tools

Tremain (2011) followed up this work to consider how the situation had changed after the introduction of electronic marking and online standardisation. The study looked at the factors that influence the satisfaction that examiners express about their work, and highlighted three factors underpinning examiners' intention to continue:

1. The relationship between examining work and work outside examining
2. The pressures of examining and support received
3. The incentives for examining

The study states that although there is no imminent threat to examiner retention, future threats include the increasing use of online tools, which can contribute to examiners feeling unsupported or undervalued. Job satisfaction is considered to be more important in retention than reward for the majority of jobs, with social interaction and appropriate challenge being considered particularly valuable. The adoption of online tools had contributed to a sense of isolation amongst examiners, and also made the work more routine - although the reliability of marking has actually increased as a result.

A further study (Tremain, 2012) also set out to evaluate how specific factors involved in online marking & standardisation contributed to examiner satisfaction. This concluded that there was no significant difference in intention to continue marking between examiners who were standardised using face-to-face or online methods. Examiners who had marked using a mixture of paper and online methods showed a very slight increase in intention to continue examining. However, it was noted that the results were confounded by the different subjects and levels of experience amongst the participants.

Variables that we may be able to influence, and how:

• Support received. By considering the different levels of support that are currently offered from the contextual model for learning (Shepherd, 2011) and identifying possible gaps, we may be able to improve the support offering for examiners in a rational way. I have already laid out some initial thoughts for this approach.
• Insight gained from examining. Making key insights from senior examiners available in a digital form which can be shared more easily online, for instance through learning management systems or webinars, could help to ensure efficient dissemination of relevant information.
• Social interaction. This is a long term goal that our organisation may want to consider for retaining examiners. Although we are increasingly unable to provide opportunities for examiners to meet in a face-to-face setting, there are possibilities for facilitating some more informal interaction around scheduled events. One of my colleagues is keen to run webinars for examiners to gain insight from senior examiners, and careful use of online chat could help to provide a better sense of community.

Any or all of these methods could be attempted, with measurement of the effect on intention to continue, and also examiner performance, being undertaken to determine effectiveness. One concern I have is that apparent failure to make a difference at first might result in a loss of enthusiasm for innovation, hence there would need to be trust established with stakeholders for future improvements. Undertaking action research alongside quantitative measurements to demonstrate a rational approach would be key to successful establishment of such trust.

References:

• Meadows, M., 2004. Measures of examiner job satisfaction as predictors of intention to continue examining. Available at: https://cerp.aqa.org.uk/research-library/measures-examiner-job-satisfaction-predictors-intention-continue-examining [Accessed December 8, 2013].
• Shepherd, C., 2011. The new learning architect, Chesterfield, U.K.: Onlignment.
• Tremain, K., 2011. Carry on examining: what predicts examiners’ intentions to continue examining? Available at: https://cerp.aqa.org.uk/research-library/carry-examining-what-predicts-examiners-intentions-continue-examining [Accessed December 7, 2013].
• Tremain, K., 2012. Carry on examining: further investigation. Available at: https://cerp.aqa.org.uk/research-library/carry-examining-further-investigation [Accessed December 7, 2013].

Architecture

Going off on a little bit of a tangent, it's time to take a look at what support is being offered to our examiners through the e-learning provision, and how it could be improved. Shepherd (2011) provides a contextual model for learning, based on four contexts: experiential, on-demand, non-formal, and formal; and two perspectives: top-down and bottom-up.

• Experiential learning is learning from as opposed to learning to. We have to be actively engaged with our task and - one hopes - reflecting on our successes and failures. There are a great wealth of lessons that can be learned while examining - Meadows (2004) and Tremain (2011) report that examiners consistently cite the insight gained from examining as one of the key benefits. Considering how to support or encourage such reflections might help to improve examiner performance.
• On-demand learning is learning to perform a particular task and acquire the necessary knowledge, at the point of need or 'just-in-time'. Depending on how far in advance our examiners access the learning materials, they could be regarded in this fashion, although they are probably better used in the following category.
• Non-formal learning is also learning to but with a more relaxed time frame, where employers take steps for employees to be prepared in the medium to long term, and is sometimes labelled as 'just-in-case' learning to set it apart from on-demand. It is distinguished from formal learning below by not being packaged as a formal 'course', which will be the case for this intervention, although there may be something to be said for carefully considering how the materials are to be presented to examiners.
• Formal learning is defined by clearly set learning objectives, a means of assessment, and usually some kind of qualification. We definitely don't offer a qualification for learning to examine (perhaps some would say we should?!), assessment would be somewhat laborious unless it were to be done covertly through completion, and the learning objectives are difficult to define. It is far easier here to think in terms of business objectives - 'There are no learning metrics, only business metrics' (Cross) That being said, it will be worth considering what examiners will expect to see and make sure that objectives are clearly stated.
• Top-down learning is aligned with employers' objectives, and is intended to ensure performance is not left to chance, and sets out to ensure that the requisite skills and knowledge can be acquired to do so. Our organisation is responsible for ensuring that results are delivered on time and accurately, with severe penalties possible for failing to do so.
• Bottom-up learning occurs because of employees' motivation to perform effectively. In addition to the motives around improved insight, examiners are drawn to the extra pay for examining, and improved promotion prospects in their teaching roles (Meadows, 2004).

Moving into specifics from Shepherd's model, there are several components that are either currently present in our business model, introduced explicitly through the e-learning provision, and some that perhaps should be there. The existing provisions focus exclusively on the top-down perspective:

• There are already performance appraisals built into our way of working (experiential);
• A help-desk is provided through our Contact Centre (on-demand);
• Examiners typically receive a certain amount of on-job training through contact with their supervisors (non-formal);
• For the live pilot, examiners will be receiving classroom courses (formal);
• The majority of users will have access to the rapid e-learning materials (non-formal). Note that I avoid referring to our e-learning as self-study e-learning (a formal intervention), which by definition should provide 'instant and individualised feedback', something that is far beyond the scope of our planning.

If we open ourselves up to the full range of possibilities from Shepherd's model, I would be tempted to add the following methods:

• Webinars could be used to convey a lot of the material and briefing that might take place in a face-to-face context, without examiners having to travel to a central location, and allow for some questions & answers
• Online video could be a powerful tool for engaging examiners with the task at hand, especially if delivered by senior examiners involved in the pilot. The message would have to be particularly clear, relevant and to-the-point, requiring serious consideration before asking for this intervention.
• Performance support materials could be leveraged for contact centre staff and senior examiners, drawing on key lessons from the live pilot.
• Forums could be used amongst contact centre staff to post common questions from examiners; there could also be forums available to examiners to make common or emergent solutions available.
• Of course I would love it if someone other than myself found a reason to keep a blog ...

That's all for now I think, I'll look at specific applications in future posts....

References:

• Meadows, M., 2004. Measures of examiner job satisfaction as predictors of intention to continue examining. Available at: https://cerp.aqa.org.uk/research-library/measures-examiner-job-satisfaction-predictors-intention-continue-examining [Accessed December 8, 2013].
• Shepherd, C., 2011. The new learning architect, Chesterfield, U.K.: Onlignment.
• Tremain, K., 2011. Carry on examining: what predicts examiners’ intentions to continue examining? Available at: https://cerp.aqa.org.uk/research-library/carry-examining-what-predicts-examiners-intentions-continue-examining [Accessed December 7, 2013].