Looking the other way

Image: freeimages

I've done quite a lot of searching through the research library to summarise some of the key findings of my colleagues, but what does everyone else think? There are other exam boards, and undoubtedly many independent researchers in the field of GCSE and A-level examining, so I've done a search for some alternative viewpoints on the matter.

Marking and cognitive psychology

How do examiners actually go about the process of marking, and does it actually matter? Greatorex & Suto (2006) looked at cognitive approaches taken by examiners when undertaking marking, and identified five distinct approaches that were used - rarely in isolation. The approaches identified were matching, scanning, evaluating, scrutiny and no response. These were related to the 'System 1' (quick, associative) and 'System 2' (slow, rule-governed) thought processing models described by psychologists (Kahneman & Frederick, 2002).

The study involved having examiners mark papers, and 'think out loud' about the approach they were taking. When marking short-form answers that could be easily distinguished by single words or numbers, examiners used the 'matching' approach (System 1) to quickly assign marks by pattern recognition. Some longer answers could be marked through the use of 'scanning' to pick either key words (System 1, pattern recognition); or distinct phrases or stages of calculations (System 2, semantic processing). For more detailed answers, examiners moved to the 'evaluating' approach to assess the candidate's response, usually drawing on a variety of sources, and compare these to their own knowledge and the mark scheme (entirely System 2). Where responses deviated noticeably from the mark scheme, examiners would engage in 'scrutinising' to identify if the response was worthy of credit, for instance an unexpected but valid response; this approach also draws entirely on System 2. In the case of 'no response', examiners would use a simple System 1 approach to check that material has not been written elsewhere.

The researchers then went on to analyse how frequently the different approaches were used in different subjects. There was a marked difference between Mathematics and Business Studies papers: Mathematics responses called for a high level of matching, with slightly less evaluation, and relatively small amounts of scanning and scrutinising; Business Studies drew heavily on evaluating, with matching as the secondary approach, and small amounts of the other approaches. Most importantly, the study showed that the different approaches were used across multiple subjects.

There was notably no relation between marking strategy and marking reliability - multiple approaches could be equally valid and successful. There was also no significant difference between marking approaches for novice and experienced markers. Senior examiners went on to suggest that new examiners could benefit from some explicit advice being given to new examiners about their approach to marking, possibly with screen recordings overlaid with commentary. The researchers also noted that there did not seem to be any difference in cognitive approach between paper-based marking and on-screen marking, although this had yet to be confirmed by direct study.

References:

• Greatorex, J. & Suto, W.M.I., 2006. An empirical exploration of human judgement in the marking of school examinations. In International Association for Educational Assessment Conference. Singapore. Available at:http://iaea.info/documents/paper_1162a2471.pdf
• Kahneman, D. & Frederick, S., 2002. Representativeness revisited: Attribute substitution in intuitive judgment. In T. Gilovich, D. Griffin, & D. Kahneman, eds. Heuristics and biases:  The psychology of intuitive judgment. New York,  NY, US: Cambridge University Press, pp. 49–81.