PISA: leaning tower?

Image: MeiTeng via freeimages.com

The PISA tests attract a great deal of attention across the world, having quickly become the most influential rankings in international education, with students from around the world taking part every three years (Coughlan, 2013 a).

Are they a fair test, or even a necessary one? Andreas Schleicher of OECD is adamant that the tests represent a fair way of assessing what comes out of a country's education system, rather than simply how much money is poured into it (Coughlan, 2013 b). He believes it should serve as a wake up call  to our complacency; are we giving students a false sense of security that will quickly be destroyed in the face of global (or even local) competition? Even in his own country of Germany, the PISA tests acted to damage faith in the system as the results did not match up with the perceptions of the nation; while in Italy, the tests suggested a real inequality in national test scores between north and south. In the UK, the relentless rise in top grades achieved at A-level seems to have stalled in recent years (Coughlan, 2014) - perhaps this is an indication that the government have been forced to rein in their confidence?

The UK government has certainly been taking the stagnation in PISA ratings seriously (Coughlan, 2013 a), with changes made to the curriculum and attempts to give extra financial support to poor performing pupils. However, the government's reaction has been criticised by teachers, as it does not place importance on their professionalism. They point out that higher performing countries make their achievements by letting schools and teachers collaborate, instead of having ministers try to make judgements based on assumptions about market forces. Similar concerns have been raised about reactions in the US (AFTHQ, 2013), as both government and pressure groups try to apply forceful, top-down solutions to quickly bolster the test scores, instead of looking deeper into the data and trying to emulate the methods and philosophies of the top-performing nations.

We always seem to come down to how competitive we can be as a society when reacting to these test scores, but perhaps we are ignoring a deeper and growing trend towards cooperation, with individuals taking advantage of technology to share knowledge in ways previously considered impossible, and often in ways that will actually benefit their competitors as well (Price, 2013).

References:

• AFTHQ, 2013. What Does the PISA Report Tell Us About U.S. Education?, Available at: http://www.youtube.com/watch?v=hf9UVg-TdH0&feature=youtube_gdata_player[Accessed August 16, 2014].
• Coughlan, S., 2013 a. UK “stagnates” in global school test. BBC. Available at: http://www.bbc.co.uk/news/education-25187997 [Accessed August 16, 2014].
• Coughlan, S., 2013 b. How Pisa became the world’s most important exam. BBC. Available at: http://www.bbc.co.uk/news/business-24988343 [Accessed August 16, 2014].
• Coughlan, S., 2014. A-levels dip as degree places rise. BBC. Available at: http://www.bbc.co.uk/news/education-28772974 [Accessed August 16, 2014].
• Price, D., 2013. Open: How we’ll work, live and learn in the future, England: Crux Publishing Ltd.

Method in the madness?

Image: freeimages

So I've got my research plan ready, and now just need some final feedback from my supervisor and an ethics check for the sound research plan that I've put together. It is sound - I'm sure, aren't I? Having spent a long time reading and re-reading books on research methods, constructing, de-constructing and re-constructing my world view, I should be able to put together a set of research questions that covers all bases for both my own project and the needs of my company. Having said that, I can't help but wonder if I've caught all the main areas; if I haven't left out some critical parts under the time pressure to get a final set of questions 'out there' in time for the project going live.

So how am I going about this? I've been balancing the needs for my own learning by phrasing many of the questions in an open-ended manner that should draw on individual experiences, along with some questions aimed at evaluation of how the examiners actually use the resources. I'll also be inferring some evaluation from the kind of responses that examiners give to the open-ended questions. However there does seem to be a need for more concerted evaluation now that I look back over the questions, so I'm considering how I might go about drawing out some extra data without imposing too much on busy people!

Looking the other way

Image: freeimages

I've done quite a lot of searching through the research library to summarise some of the key findings of my colleagues, but what does everyone else think? There are other exam boards, and undoubtedly many independent researchers in the field of GCSE and A-level examining, so I've done a search for some alternative viewpoints on the matter.

Marking and cognitive psychology

How do examiners actually go about the process of marking, and does it actually matter? Greatorex & Suto (2006) looked at cognitive approaches taken by examiners when undertaking marking, and identified five distinct approaches that were used - rarely in isolation. The approaches identified were matching, scanning, evaluating, scrutiny and no response. These were related to the 'System 1' (quick, associative) and 'System 2' (slow, rule-governed) thought processing models described by psychologists (Kahneman & Frederick, 2002).

The study involved having examiners mark papers, and 'think out loud' about the approach they were taking. When marking short-form answers that could be easily distinguished by single words or numbers, examiners used the 'matching' approach (System 1) to quickly assign marks by pattern recognition. Some longer answers could be marked through the use of 'scanning' to pick either key words (System 1, pattern recognition); or distinct phrases or stages of calculations (System 2, semantic processing). For more detailed answers, examiners moved to the 'evaluating' approach to assess the candidate's response, usually drawing on a variety of sources, and compare these to their own knowledge and the mark scheme (entirely System 2). Where responses deviated noticeably from the mark scheme, examiners would engage in 'scrutinising' to identify if the response was worthy of credit, for instance an unexpected but valid response; this approach also draws entirely on System 2. In the case of 'no response', examiners would use a simple System 1 approach to check that material has not been written elsewhere.

The researchers then went on to analyse how frequently the different approaches were used in different subjects. There was a marked difference between Mathematics and Business Studies papers: Mathematics responses called for a high level of matching, with slightly less evaluation, and relatively small amounts of scanning and scrutinising; Business Studies drew heavily on evaluating, with matching as the secondary approach, and small amounts of the other approaches. Most importantly, the study showed that the different approaches were used across multiple subjects.

There was notably no relation between marking strategy and marking reliability - multiple approaches could be equally valid and successful. There was also no significant difference between marking approaches for novice and experienced markers. Senior examiners went on to suggest that new examiners could benefit from some explicit advice being given to new examiners about their approach to marking, possibly with screen recordings overlaid with commentary. The researchers also noted that there did not seem to be any difference in cognitive approach between paper-based marking and on-screen marking, although this had yet to be confirmed by direct study.

References:

• Greatorex, J. & Suto, W.M.I., 2006. An empirical exploration of human judgement in the marking of school examinations. In International Association for Educational Assessment Conference. Singapore. Available at:http://iaea.info/documents/paper_1162a2471.pdf
• Kahneman, D. & Frederick, S., 2002. Representativeness revisited: Attribute substitution in intuitive judgment. In T. Gilovich, D. Griffin, & D. Kahneman, eds. Heuristics and biases:  The psychology of intuitive judgment. New York,  NY, US: Cambridge University Press, pp. 49–81.

Reliability of public examinations

Image: freeimages

What makes for a reliable exam series? And who actually knows or cares?

A community effort?

Baird, Greatorex and Bell (2004) set out to study the effect that a community of practice could have on marking reliability - their study concluded that discussion between examiners at a standardisation meeting had no significant effect on marking reliability, therefore the effect of a community of practice was called into question. I have to challenge this conclusion, because the artificial and synchronous conditions of a standardisation meeting most definitely do not represent the potential for the asynchronous nature of a community of practice. The authors didn't take this into account - interesting that they actually cited Wenger in their references. Later on in the paper they actually seem to acknowledge that tacit knowledge already acquired in a community of assessment practice might actually explain why the format of standardisation meetings seemed to have no significant effect - this seems more in line with the effect that I would expect a community of practice to have.

What do these people know anyway?

Later research by Taylor (2007) sought to investigate public perceptions of the examining process. The study involved a range of participants - students, parents and teachers, using a range of interview techniques, and looking at issues such as how papers are marked, the reliability and quality of marking, and the procedure for re-marks. The level of awareness about these issues varied somewhat, but it seemed very few people (even teachers) had full knowledge of the entire process.

There seemed to be a perception among students and parents that several examiners would mark each script (p.6) and arrive at a final mark through consensus, while teachers generally felt that a single examiner marked each paper, but this was based on perception rather than firm knowledge. Students and parents did not seem to have any real knowledge of how examiners might arrive at a mark, and even teachers were not aware of the hierarchical method, although they agreed that it seemed sensible when it was explained to them. All groups believed that the system would work better with multiple examiners, although they did acknowledge that this might be unrealistic given time and financial constraints. When questioned about the possible merits of having multiple examiners mark a single paper, examiners commented that any gains would be minimal in comparison to the present hierarchical system, and the cost would be prohibitive.

Members of the public seemed to have a better awareness of the concept of reliability (p.7), with teachers having a similar perception to examiners about the potential for marks between examiners to differ within a band of ability. Students and parents seemed to understand the potential for marks to differ, although their expectations were more optimistic than the real situation. There was much less understanding of how quality of marking was assured (p.8), with perceptions varying from an assumption that there was no checking at all. to some people believing that far more scripts were checked than is the case. All parties agreed that quality checking was desirable, and examiners appeared to support the current system. Re-marking was more common knowledge to all the participants (p.10), although there was little knowledge of the precise system used.

The report considered whether attempts to increase public understanding of the exam system would promote public trust - although some believe that greater transparency might actually invite criticism, other literature seems to suggest that revealing the workings of the system would lead to more realistic expectations and improved engagement, rather than a focus on failings. Establishing a clearer link between understanding of the exams process and public confidence was suggested for future research.

More on reliability

Chamberlain (2010) set out to build on the research into public awareness of assessment reliability - which also ties in well with some of the points raised by Billington (2007) about whether the general public are acting on good information or misinformation. The research into public awareness used focus groups as a means of drawing out understanding, as it bypasses some of the possible problems of researcher bias (since the researchers were AQA employees, and can help foster a collaborative environment to trigger the sharing of opinions and ideas (p.6). One remaining source of bias was that the groups were small and were composed primarily of people with a particular interest in the exams.

Research with several groups of people, including a number of teachers and trainee teachers, suggested little general understanding of the concept. Secondary teachers had more understanding, often due to exposure to requesting re-marks for their pupils. Promisingly, most of the participants cared enough to indicate they would like to be better informed about the overall process of examinations, although there was no support for any quantification of reliability - most people already seemed to accept that there would inevitably be some variance in how marks were awarded, but placed trust in examiners to act as professionals. The exception to the rule is when a very public failure occurs, but even in this case attention rapidly fades once the problem seems to be 'fixed', with little real gain in understanding. The video below is one attempt to get some understanding out into the public domain:

Making The Grades - A Guide To Awarding

References:

• AQA Education, 2013. Making The Grades - A Guide To Awarding, Available at:http://www.youtube.com/watch?v=S9orrawQQII&feature=youtube_gdata_player[Accessed April 5, 2014].
• Baird, J., Greatorex, J. & Bell, J.F., 2004. What makes marking reliable? Experiments with UK examinations. Assessment in Education: Principles, Policy & Practice, 11(3), pp.331–348. Available at: http://www.tandfonline.com/doi/abs/10.1080/0969594042000304627 [Accessed December 8, 2013].
• Billington, L., 2007. Public trust and high stakes assessment. Available at: https://cerp.aqa.org.uk/research-library/public-trust-and-high-stakes-assessment [Accessed March 27, 2014].
• Chamberlain, S., 2010. Public perceptions of reliability. Available at: https://cerp.aqa.org.uk/research-library/public-perceptions-reliability [Accessed March 29, 2014].
• Chamberlain, S., 2012. Qualification users’ perceptions and experiences of assessment reliability. Available at: https://cerp.aqa.org.uk/research-library/qualification-users-perceptions-and-experiences-assessment-reliability [Accessed March 29, 2014].
• De Moira, A.P. et al., 2002. Marking consistency over time. Research in Education, 67(-1), pp.79–87. Available at: http://dx.doi.org/10.7227/RIE.67.8 [Accessed December 8, 2013].
• Pinot de Moira, A., 2013. Features of a levels-based mark scheme and their effect on marking reliability. Available at: https://cerp.aqa.org.uk/research-library/features-levels-based-mark-scheme-effect-marking-reliability [Accessed March 29, 2014].
• Royal-Dawson, L. & Baird, J.-A., 2006. Is teaching experience necessary for reliable marking? Available at: https://cerp.aqa.org.uk/research-library/teaching-experience-necessary-reliable-marking [Accessed December 8, 2013].

Trust

Image: freeimages

In December 2011, a scandal emerged from the exposure of corrupt practices by two senior examiners from an examination board (Newell, 2011; Orr, 2011; Watt, 2011), who were captured on video telling teachers at a paid-for seminar what subjects would appear on upcoming exam papers. The ensuing public outcry and reaction from government officials led to accusations that the whole exams system was compromised by profit motives (Garner, 2011), and even calls to abolish the exam boards and replace them with a single body.

What causes examiners, with such a degree of trust placed in them, to break with the values of the education system in this way? How badly eroded is the trust in our public education system? Was this an isolated incident, or was it just a flashpoint for a growing sense of discontent with education in our country?

Four years prior to this incident, Billington (2007) reviewed the literature on trust in public institutions, with a view to building an understanding of how to look at issues of trust within examinations and examination standards. Many public institutions appear to suffer from what is described as either a 'crisis of trust' or a 'culture of suspicion', characterised by an expressed lack of faith in the institution - although this does not always equate to a lack of trust in individual professionals within that system, particularly when there is an immediate need for their skill.

There are also distinct shifts in the way that institutions and members of the public relate to one another, largely relating to the impact of information and communications technology (Billington, 2007, p.2), which alter perceptions of and demands for equality. Professionals have access to increasingly powerful tools and knowledge designed for their use, whilst the general public also have access to a wider body of information - and misinformation. People often feel the need to gain a little control over professionals whose power affects their lives, and they will attempt to use any tools at their disposal to get a sense of equality. When there is a huge deal of accountability data available, members of the public may use this accountability as a substitute for trust, which can lead to negative consequences.

In the case of public examinations, the consequences have often been that schools gravitate towards exam boards that offer less demanding specifications. There was a significant move towards the International GCSE and International Baccalaureate qualifications without any formal recognition from government bodies, which seems indicative of an erosion of trust. Education scandals in previous years had already brought a higher public awareness of examination standards. There has also been an increased politicisation of education, with an apparent mismatch between the goals of the government and the exams watchdog QCA being at odds with one another. It seems then that the media coverage of the cheating examiners has sparked a strong response to a growing sentiment of suspicion.

The qualifications environment in England has become increasingly similar to one of a free market economy (Jones, 2011); language such as 'gold standard' and 'currency of qualifications' suggest by metaphor that our understanding of and behaviour towards qualifications is strongly shaped by market philosophy. Allowing market forces to enter an area of public service can affect perceptions of trust - Billington (2007, p. 6-7) mentions the medical profession as an example of public trust being undermined by suspicion about the motives of those who represent the profession. When qualifications are treated as currency, we risk losing sight of the real educational value - just as actual currency no longer has any real relation to a physical 'gold standard' any more.

I realise that I am starting to veer off to a new matter entirely, so I'll end this blog post here!

References:

• Billington, L., 2007. Public trust and high stakes assessment. Available at: https://cerp.aqa.org.uk/research-library/public-trust-and-high-stakes-assessment [Accessed March 27, 2014].
• Garner, R., Profit motive has created corrupt education system, say teachers. The Independent. Available at: http://www.independent.co.uk/news/education/education-news/profit-motive-has-created-corrupt-education-system-say-teachers-6274531.html [Accessed March 27, 2014].
• Jones, B., 2011. Regulation and the qualifications market. Available at:
  https://cerp.aqa.org.uk/research-library/regulation-and-qualifications-market [Accessed March 29, 2014].
• Newell, C., 2011. Exam boards: “We”re cheating, we’re telling you the question cycle'. Telegraph.co.uk. Available at:
  http://www.telegraph.co.uk/education/secondaryeducation/8940799/Exam-boards-Were-cheating-were-telling-you-the-question-cycle.html [Accessed March 29, 2014].
• Orr, J., 2011. Exam boards: examiners suspended in “corrupt practices” row.Telegraph.co.uk. Available at: http://www.telegraph.co.uk/education/secondaryeducation/8943300/Exam-boards-examiners-suspended-in-corrupt-practices-row.html [Accessed March 27, 2014].
• Watt, H., 2011. Exam boards: how examiners tip off teachers to help students pass. Telegraph.co.uk. Available at:
  http://www.telegraph.co.uk/education/secondaryeducation/8940781/Exam-boards-how-examiners-tip-off-teachers-to-help-students-pass.html [Accessed March 29, 2014].

Re-affirming the project outline

My project outline has remained a little vague, so I thought I would try to add more details to it...

We currently provide secondary qualifications for 14 - 19 year olds. This work is dependent on a network of some 35,000 teachers and other experts to help set and mark examinations, with technology playing an increasingly pivotal role to ensure fast and scalable transfer of marks. We are continually exploring new technologies for mark capture and transfer to ensure the best possible service for candidates.

Trialling and adopting a new marking technology requires training provision for large numbers of examiners. Previous technology adoptions have initially been dependent on government funding for their initial success, but this funding is no longer available. Training provision has increasingly moved towards the creation of online software demonstration videos and interactive simulations, now hosted on a secure Learning Management System (LMS).

We are currently piloting a new marking technology with a very small number of examiners who have received face-to-face training, and are now looking to move to exclusively online training as soon as possible. With previous technology adoptions the online provision has been developed largely through internal discussion after face-to-face training and released without a live test for examiners. For this project, the online learning materials are being developed alongside the first live pilot of the technology, with an opportunity for early testing and feedback.

We will be using action research methods to inform improvements to the online learning materials and identify additional support methods prior to general release. This research will be expanded for the live use of the software during the summer examination series, with a view to providing both evaluation of success and action research for practitioner development.

Five questions

Image: freeimages

I'm currently working through some questions for framing qualitative research (Mason, 2002), which I hope will help me to better frame my research.

The social 'reality': Your Ontological Perspective

What is the nature of the phenomena, or entities, or 'social reality' that I wish to investigate?

Elements from Table 1.1 that appeal:

• People, social actors, motivations, identities, cultural or social constructions
• Experiences, development, behaviours, interactions, social processes
• Institutions, markets, societies, organisation, connectedness, multiple realities or versions + tribes, networks

These are more relevant to my overall world-view, so I will generate an additional set specific to the current project:

• People, understandings, perceptions,attitudes, thought
• Experiences, development, actions, interactions, situations, rules
• The 'material', groups, organisation

Knowledge and Evidence: Your Epistemological Position

What might represent knowledge or evidence of the entities or social 'reality' that I wish to investigate?

Experiences of users will allow me to link my design approach to how people perceive the materials and support framework - affects my ability to rationally design or re-design materials and approaches in response to user feedback.

The research would benefit from some objective measures of performance to show that knowledge about systems and procedures has successfully transferred, rather than asking for user ratings - allows for genuine accountability in our training approach.

Your Broad Research Area

What topic, or broad substantive area, is the research concerned with?

This research topic is concerned with understanding what experiences examiners have when using online resources and support to adopt new software for marking and standardisation, and any different procedures that must be adopted. My focus is on interpreting results in such a way as to allow continuous improvement of the design of such materials, and to understand how much effect the strategy has on learning, as evidenced through performance. I do not have control over the precise methods for measuring performance, but am able to access such data.

Your Intellectual Puzzle and Your Research Questions

What is the intellectual puzzle?

How do I show that my learning intervention has had the desired impact, and how can I rationally design better approaches in future, or re-design to off-set any shortcomings?

What do I wish to explain or explore?

I wish to explain how the support approaches link to performance and attitudes among the examiners, and explore how to improve both in the future.

What type of puzzle is it?

This is a causal / predictive puzzle.

Your Research Questions

What are my research questions?

For gaining insights from examiners taking part in the live pilot I'm thinking along the lines of:

• Describe your experiences of using the software to carry out standardisation of marking
• Describe your experiences of using the software to record and submit your marks
• Describe your experiences of communicating with your senior examiner during the marking period

For improving on my online materials:

• Describe your experience of using the online resources to prepare you for marking
• Describe your experience of using the online resources and printable materials during the marking period

More of these to come - from previous experience I find that it's best to mull over these and look at what I've written down with a fresh perspective later.

Your Aims and Purpose

What is the purpose of my research? What am I doing it for?

The purpose of this research is to ensure the timely and accurate delivery of high quality marking for candidates sitting national examinations. The research will be done on behalf of my employer, to benefit examiners whilst delivering the marking and help to maintain motivation through consideration of the support offered. By achieving these objectives it is also expected that examiner performance will improve, which will be judged through the assessments from senior examiners and staff. The timeliness and accuracy of results could also be interrogated relative to expected deadlines and the number of examiners stopped or re-marked.

References

• Mason, J., 2002. Qualitative researching, London: SAGE Publications.