Time for a numbers game?

Image: freeimages

Following on from my last blog post, I'm re-treading the sequence of reading from our Research Methods module to get my bearings again, and I'm coming back to the question of qualitative vs quantitative research. While I strongly identified with the Action Research methodology on my last project, it's worth deliberately opening up my mind to new possibilities, especially as there will be strong interest in some kind of numerical data from colleagues and external auditors if we are questioned on our approach.

So before I start to choose which quantitative disciplines I might wish to draw on, I'll look at the key aspects of quantitative research, consider those that appeal to me, and those I wish to avoid.


Concern with theory

Relating my findings to theory will be helpful to ensure some kind of tethers to related work, but there's a danger of getting obsessed with reproducibility and control here. When you're moving into the realm of on-demand learning, you can't guarantee learning outcomes, nor indeed that learners will even access the materials or activities that you produce for them. Newby (2010, p.96) acknowledge the limitations for educational researchers trying to identify pattern and control influences, as they are only able to view a small part of the overall education system. I would prefer to think in terms of Praxis (Wheeler, 2013), which requires practitioners to consider how closely their practice overlaps with the theories they identify with.

Concern with proof

Here lies one of the real problems for educational research - although I understand that establishing proof would give greater peace of mind, the complexity and ambiguity of the situation makes this extremely difficult:

• The situation I face will not be the same as another practitioner does, even if our verbal descriptions of it seem similar to the untrained eye
• The next situation that I (and the learners) face will not be the same as this one, even if it's 'just another e-marking system'
• The time needed to establish proof would be completely at odds with the time pressures for the project, where the learning is 'on-demand'.

The best that I can hope for is to show that using theories to guide my design leads to a dependable business outcome, and that particular methods or techniques are better suited to my situation.

Identification of variables

This is one of the key aspects of quantitative theory that I see as helpful. Although my control over most independent variables involved will be limited to say the least, it will definitely be helpful to at least make some systematic efforts to identify variables in the design of materials that may be having an effect, and to measure any dependent variables which are of interest. Our particular concerns would be the performance of examiners, and intention to continue based on their experiences. Attempting to correlate these with participation in the different aspects of the support might yield useful insights into which components have succeeded, but this would have to be linked to effective practice in design.

Simply saying that an approach should be abandoned because it doesn't seem to have an effect in this situation would be potentially misleading without some understanding as to why. Creswell (2009, p.49) refers to confounding variables (e.g. discriminatory attitudes) that can come into play, which I have had some experience of when trying to introduce online learning methods in the past. Participants who are negative about the use of the tools go to great lengths to discredit them when given the opportunity to do so, whilst the majority of participants actually acknowledge a positive effect.

Conclusion

This project will benefit from the use of some quantitative approaches to analysing data about examiner performance and intention to continue, but these will need to be paired effectively with qualitative methods to understand what dependent variables relating to the choice and design of approaches might be influencing the outcomes. My next blog post will focus on the type(s) of quantitative research methods might be useful, followed by a look at rational design approaches for the learning provision.

References:

• Creswell, J. W. (2009). Research Design: qualitative, quantitative and mixed methods approaches. (3rd edition) Sage.
• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.
• Wheeler, S. (2013). Praxis makes perfect. Learning with e's [blog] 31 October. Available at: <http://steve-wheeler.blogspot.co.uk/2013/10/praxis-makes-perfect.html>

The nature of academic publishing

Image: forwardcom via freeimages

Building up towards my reflective assignment, I thought it might help to draft this in sections, according to particular areas of interest throughout the module. First stop is publishing...

One of the assumptions that I am interested in challenging is the area of academic publishing. We tend to regard the world of academic journals as ‘an open and public forum’ (Colquhoon, 2006) that is created for the common good, but a growing number of people are beginning to question if this holds true, especially in the age of open web content. Wheeler (2011a, b, c) has written a number of blog posts criticising the traditional system of academic publishing, noting that:

‘It’s a hierarchy that rarely changes.’

Furthermore, the hierarchical nature of the publishing world is often self-defeating in terms of securing the widest possible audience:

‘...many of the top, elite journals we are expected to publish in are in fact read by a very small percentage of the community the research is intended to reach.’

From considering my own worldview, I have noted the restrictions that hierarchical institutions place on the education system, particularly when combined with the solutions offered by market organisations (Hobson, 2012; Ronfeldt, 2012b). Potential disruptive alternatives may yet be found in the emerging network sector predicted by Ronfeldt (2012a), or the peer-to-peer ideology (Bauwens, 2009), although sustainable business models are yet to emerge. It should be noted that there are dangers inherent in the misuse of data for either solution. Newby (2010) warns of ‘intellectual imperialism’ when researchers are able to claim sole ownership of research data, discounting the rights of those involved in the study. At the other extreme of open web publishing, Lanier (2006a, b; 2012) warns of the dangers of ‘digital Maoism’, whereby all ownership and authorship of published data is lost, and transferred to control of the crowd.

References:

• Bauwens, M. (2009). Open Process academic publishing: towards the peer production of academic peer review. P2P Foundation's blog [blog] 15 September. Available at: <http://blog.p2pfoundation.net/open-process-academic-publishing-towards-the-peer-production-of-academic-peer-review/2009/09/15>
• Colquhoon, D. (2006). Research Methods in Education Contexts. University of Hull
• Hobson, J. (2012). The problems with educating a quadriform society. Thinking and learning in the digital age [blog] 7 August. Available at: <http://jimmyhob.blogspot.co.uk/2012/08/the-problems-with-educating-quadriform.html>
• Lanier, J. (2006a). Digital Maoism: The hazards of the new online collectivism. Edge.org [online] 30 May. Available at: <http://www.edge.org/3rd_culture/lanier06/lanier06_index.html>
• Lanier, J. (2006b). Beware the online collective. Edge.org [online] 25 December. Available at: <http://www.edge.org/3rd_culture/lanier06/lanier06.2_index.html>
• Lanier, J. (2012). Are we at the beginning of the rise of post human machine intelligence? [Video online] Available at: <http://dpcloud.co/v3/enterprise/learningtech/index.php?option=com_dpcloud&v=play&p=7117&u=learningtech&cat=Keynotes&Itemid=2>
• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.
• Ronfeldt, D. (2012a). TIMN in 20 minutes: a video presentation. Visions from two theories [blog] 23 May. Available at: <http://twotheories.blogspot.co.uk/2012_05_01_archive.html>
• Ronfledt, D. (2012b). Q’s & A’s about “TIMN in 20 minutes” (4th of 7): creation of a new sector? Visions from two theories [blog] 29 August. Available at: <http://twotheories.blogspot.co.uk/2012/10/qs-as-about-timn-in-20-minutes-4th-of-7_29.html>
• Wheeler, S. (2011a). Sharp practice. Learning with e's [blog] 26 September. Available at: <http://steve-wheeler.blogspot.co.uk/2011/09/sharp-practice.html>
• Wheeler, S. (2011b). The open case. Learning with e's [blog] 2 October. Available at: <http://steve-wheeler.blogspot.co.uk/2011/10/open-case.html>
• Wheeler, S. (2011c). When the dam breaks… Learning with e's [blog] 18 October. Available at: <http://steve-wheeler.blogspot.co.uk/2011/10/when-dam-breaks.html>

Blog post 4.2: What could go possibly go wrong?

Originally posted to eBridge, 17 February 2013

 

Validity, Reliability, Objectivity = language for quantitative researchers

Credibility, Dependability, Confirmability = language for qualitative researchers

The classic research effects noted by Newby (2010, p.120-3) are of interest to me – I will be researching from a very participatory standpoint, as I am actually trying to influence people’s behaviour through my interventions. The best way to balance out this out would seem to be by adopting approaches such as triangulation to give more credibility to findings. As a learning & development practitioner working within an online community, am I actually moving more into the realm of critical research perhaps?

Reading through Colquhoon (2006), I’m struck by the similarity between the language used for reliability and validity, compared to the discussion of reliability and validity in the context of assessment, as referred to by Race (2010, Ch.4) and Crisp (2007, Ch.19). The pre-occupation of most educational institutions has been with stability of a bank of testable items for norm-referenced assessment. However, one has to consider how long this state of affairs can continue with the economy so unstable and all our assumptions about learning and education being questioned? (Wheeler, 2013)

References:

• Colquhoon, D. (2006). Research Methods in Education Contexts. University of Hull
• Crisp, G. (2007). The e-Assessment Handbook. Continuum
• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.
• Race, P. (2010). Making Learning Happen: A Guide for Post-Compulsory Education. 2nd ed. Sage.
• Wheeler, S. (2013). Learning is changing. Learning with e’s [blog] 17 February. Available at: <http://steve-wheeler.blogspot.co.uk/2013/02/learning-is-changing.html> [Accessed February 2013]

Blog post 3.2: Use and abuse

Originally posted to eBridge, 13 February 2013

Ironically, ignorance is probably the most rapidly growing problem in research. In the world of Google search, we assume that if it doesn’t appear, it just isn’t worth our while. Methods and discipline in searching are as important as ever, we just have to learn to work with automated tools – distributed cognition is something that we just can’t ignore! Just as for using people’s research properly, making connections is vital for success in knowing where to look, and having people to correct us when we do fail.

Even more pervasive is the ignorance born of misunderstanding – when we are inundated with so much readily available information, we often choose not to ‘learn’ things properly, trust information without thinking (Wheeler, 2012), or we underestimate the importance of ensuring real expertise amongst our teachers. With the ever-growing library of information on the world’s computers, it is easy for learners (and who knows, even teachers) to be misled. The research of Ingersoll (1999, 2005) suggests that teachers can be stretched thinly indeed.

Selective referencing can be born of a deliberate attempt to bias the reader in favour of one’s own research, to bring disproportionate attention to colleagues, or possibly just a symptom of ignorance. The final, malicious misuse of research is of course to back up a false claim, or to support an opinion without full consideration of the facts.

References: 

• Ingersoll, R.M. (1999). The Problem of Underqualified Teachers in American Secondary Schools. Educational Researcher, 30(1), 21-22.
• Ingersoll, R.M. (2005). The Problem of Underqualified Teachers: A Sociological Perspective. Sociology of Education, 78(2), 158-178.
• Wheeler, S. (2012). Lies, damned lies, and the internet. Learning with e's [blog] 26 April. Available at: <http://steve-wheeler.blogspot.co.uk/2012/04/lies-damned-lies-and-internet.html> [Accessed February 2013]

Blog Post 1.1: Public vs. Private

Originally posted to eBridge: 3 February 2013 

In the Research Methods reader, Colquhoon (2006, Ch.1) brings up the subject of how formal research is undertaken in 'an open and public forum', but I have to question just how open the world of academic publishing really is. Sure, there is the sense of authority from having the work reviewed rigorously by other academics, but it's then published in pay-to-subscribe journals that are usually not accessible to the general public, even if they knew what they were looking for. Despite having access to Athens, I often find I'm more successful using Google Scholar for tracking down something useful, and my Athens account sometimes doesn't get me access to the material of certain publishers that seem to be on the approved list.

Is there a different way of publishing research? Wheeler (2011) notes that a number of academics are joining a revolt against pay-to-subscribe journals. Perhaps in time this might become more than just a trickle?

References:

• Colquhoon, D. (2006). Research Methods in Education Contexts. University of Hull
• Wheeler, S. (2011). When the dam breaks... Learning with e's [blog] 18 October. Available at: <http://steve-wheeler.blogspot.co.uk/2011/10/when-dam-breaks.html> [Accessed February 2013]