Evaluating online communities

Thoughts on Ke & Hoadley (2009)

 

Important point to consider: do we expect online learning communities (OLCs) to appear spontaneously or through design? Study inherently seems to favour studies of those that are well-defined in terms of support & structure, so either well-designed or evolved.

Taxonomy of online learning community evaluations

Recognises that this is a divergent research area, attempts to categorise studies in terms of four key components:

1. Evaluation purpose
Notable distinction between proving and improving purposes – convincing organisations that the community has a value at all, versus looking for ways of systematically enhancing the interactions within it. In my case the interactions don't yet amount to a community, so the emphasis is on making interactions sustainable and of benefit to participants.

2. Evaluation approach
Approaches were sometimes summative, usually for proving, or formative for improving, sometimes with elements of both of these. There are also the participatory and responsive approaches – a choice of whether to include participant evaluation or not. Oliver (2000) and Patton (1997) are cited as primary references here. My approach will be based entirely on participant responses, with a view to formative evaluation of the interactions.

3. Measures for evaluation
Outcome vs process measures. The outcome view looks at the community as a static system, evaluating the raw technical set-up of the environment and the learning outcomes. Process evaluation takes an in-depth look at the factors that facilitate or impede learning within the system. My study will need to focus on the process, with a possibility for pairing this up with outcome evaluations from colleagues.

4. Evaluation techniques
The authors seem to use the term objective in place of quantitative; they also refer to qualitative and mixed-method approaches. An important distinction between the two main forms is made – objective approaches deliberately remove context from the data, focusing on what could be directly comparable between other studies. Qualitative studies allowed for more direct insights for the learning processes within a community. I will be focusing on qualitative approaches, although there is potential for identifying the best factors to use in future studies for building up an objective measure of communities in future.

Conclusions

This gives a very good critical analysis of the factors at play in evaluating OLCs, and can serve as a guide point relating to the higher level discussions of Newby (2010), Cresswell (2009) and Colquhoon (2006). The authors also point out a good number of shortcomings in current research. Partly these are due to the constraints of researchers performing their studies for their own purposes, rather than to sit conveniently into the wider body of research. They also point out that the offline interactions between participants play a large role in the actual learning process, and these are very difficult to find any record of. There are also no studies that show how a community has evolved over time.

Long term goals of the researchers are to establish a framework for understanding OLCs, possibly towards a central theory. They make no reference to the five stage model identified by Salmon (2003); they do refer to phases of community development (Palloff and Pratt, 1999). It will be useful to pursue this systematic approach for research to inform my own understanding, and note if any similarities or contradictions with Salmon’s model emerge, which has previously been central to my understanding of online learning interactions.

Reference:

• Colquhoon, D. (2006). Research Methods in Education Contexts. University of Hull.
• Creswell, J. W. (2009). Research Design: qualitative, quantitative and mixed methods approaches. (3rd edition) Sage.
• Ke, F. and Hoadley, C. (2009). Evaluating Online Learning Communities. Educational Technology Research and Development, 57(4), pp.487-510.
• Newby, P. (2010). Research Methods for Education. Pearson Education Limited.
• Salmon, G., 2003. E-Moderating: The Key to Teaching and Learning Online. 2nd ed. London: Routledge-Falmer.

Time to dig deeper?

My initial literature review led me to a very comprehensive article by Ke & Hoadley (2009), which was very much a centre-piece for my (admittedly still thin) literature review. Whilst I'm keen to continue using this and related articles, I'm now realising after the last few weeks of considering research methods that I need to dig a little deeper, and in a different direction. The problem is that the article focuses on evaluation research, whereas I really need some good pointers on action research, as I believe this will be of more benefit in the qualitative research project I actually intend to do.

 

One useful resource that I have come across is an overview of action research (O'Brien, 1998). Although this is a slightly older article, O'Brien covers a lot of the points made in my earlier reading, and with a slightly different perspective on how it might be used. He also gives case studies of early use of computer mediated conferencing, and how action research was utilised to give the practitioners and participants useful insights that improved practice. Most tellingly, O'Brien points out that action research in itself is becoming a vital tool for organisations that are increasingly interdependent in a climate of change, something that fits well with my worldview.

 

"If you want it done right, you may as well do it yourself"

 

Johnson (2001) offers a good overview of online interaction and communities, again slightly ahead of the review by Ke & Hoadley (2009). One point that immediately caught my eye, is the acknowledgement of attrition as being a key problem - within the workplace, simply getting people to participate in learning activities can be problematic, and online interactions even more so. There is also convergence with the works of Wenger, and of Palloff & Pratt, but again no mention of Salmon, nor of Anderson.

 

References:

• Johnson, C.M. (2001). A Survey of Current Practice on Online Communities of Practice.  Internet and Higher Education, Vol.4 pp.45-60. Available at: <http://repository.maestra.net/valutazione/MaterialeSarti/articoli/Johnson-Survey.pdf> [Accessed March 2013]
• Ke, F. and Hoadley, C. (2009). Evaluating Online Learning Communities. Educational Technology Research and Development, 57(4), pp.487-510.
• O'Brien, R. (1998). An Overview of the Methodological Approach of Action Research. [online] Available at: <http://www.web.ca/~robrien/papers/xx%20ar%20final.htm> [Accessed March 2013]

Thoughts on defining research purpose

Trying to define my research purpose effectively has required extensive reading and critical thought to make sure that my purpose is properly defined

Chapter 5: The Introduction

The Research Problem – Creswell gives some good advice that relates general writing skills to the research context. We are normally trying to convey a very complex topic, so it is important that we consider how we are going to ‘hook’ our readers into the subject, and make sure we don’t bombard them with too much information too soon. There is also the need to make sure that our work is seen as a scholarly endeavour by outlining our links to the literature.

 

Studies Addressing the Problem – Using an inverted triangle seems like a good idea for the process of zeroing in on our research problem, using the wider body of literature to provide the background. We need to acknowledge that there are limitations in the literature we draw on (otherwise there wouldn’t be anything to research) and instead look for ways that the lessons learned or methods employed could inform our own research.

Deficiencies in Past Literature – Whilst avoiding outright condemnation of the existing literature, it is important to point out where it does not (or cannot) address our particular area. For instance, whilst I am planning to use Ke & Hoadley (2009) as a primary reference point for methods, it will be useful to explore the different contexts of the studies they included in their review - how many of the studies are based in the workplace? Referring to studies of learning in the workplace could help to draw links – thinking of the inverted triangle here.

Significance of a Study for Audiences – This does have some good points about writing for particular audiences, and trying to make the study appeal to a wide variety of people. Some of my original reservations remain around the default ‘market’ mentality!  Interestingly one of the sidebars mentions organisational mentalities, and may be worth following up in its own right.

Summary – The five part introduction model should serve well in conjunction with critical thought about my world view and the literature that I will use. The writing exercises will help me to set out my thinking ahead of writing the actual paper, and hopefully draw in some useful feedback. I will make sure that I give feedback to my peers first as an incentive for them to reciprocate, and to strengthen connections amongst the group.

 

Chapter 6: The Purpose Statement

 

Creswell states that this is the most important statement in the entire study – it effectively sets out the means by which others are actually going to be assessing the worth of your research.

 

Significance and Meaning of a Purpose Statement – The purpose statement is deliberately unpicked from the separate points of research problem (the wider issues) and research questions (the fine details of the study).

 

A Qualitative Purpose Statement – Careful use of language is required here. Using the draft script to produce some initial ideas and then going back through the text as a preliminary round of feedback sounds more sensible for learning than trying to construct one from the example words given. That way I can reflect on the differences between how I normally think internally, and the way I externalise this in my writing.

 

A Quantitative Purpose Statement – Although I consider that my study will be more qualitative in nature, it is worth considering the different way that a quantitative purpose statement should be written, so that I know the difference! Considering the variables that may affect an outcome and systematically investigating them is a hallmark of a quantitative study. Although I need to consider what factors may affect communities, I will not try to quantify them at this stage. Theory is implied as being more central to a quantitative study than a qualitative one.

 

A Mixed Method Purpose Statement – Understanding the previous two types of purpose is vital for properly introducing a mixed methods study, as the reader has to be able to distinguish between them through your writing. For success, there should be a central theory, and a single qualitative phenomenon under discussion.

 

Summary – Universal techniques underpin all of the types of study. Action words must be used for outlining the purpose, whilst keeping to non-directional language (i.e. avoid trying to prove a hypothesis through interfering with data acquisition). Mentioning the strategy for the enquiry, along with the participants and site to provide context, helps a reader further discern the relevance for them.

 

Influences on the selection of a research issue (Creswell)

Research agenda – Ultimately everything needs to be paid for! Reading through some of Creswell’s assumptions on the last unit led me to challenge the assumption that research has to be targeted at the greatest number of people. This actually fits more with a market philosophy, whereas I would rather that my research fitted into the network paradigm, and also be carried out for on-going practitioner development. My organisation wants more learning to go online, but there are likely to be many different perceptions of what does and does not work. I may actually be going against accepted wisdom with my desire to research this area!

 

Research issue – From my context, online communication actually becomes a potential measure for how successful a traditional learning event has been, as opposed to one where online interactions have not been present. The Research Problem can then be further defined as outlining effective ways of evaluating training success beyond the event, fitting in with the research agenda more closely.

 

Research question – Can online learning interactions increase the effectiveness of training programs, and when are they most likely to succeed?

 

Learning from a research agenda - This section gives some very good insights on how the research agenda can shift, and acknowledges the limitations of academic journals in our learning and research.

Determining a research issue - Generation of ideas, particularly questions, is dependent on having a rich and diverse environment.  In this way we are more likely to see alternative perspectives that allow us to challenge agendas and assumptions.

Kick-starting the search for a research issue - Whilst this is not the first research project that I have conducted, I do think that I am better regarding myself as being a ‘new’ researcher as I did not undertake any of this kind of learning before my research project – everything was already funded before I started! Within my department there is not always formal research, although my colleagues did use the Delphi research method for determining the best approach to performance & talent management. I also have the benefit of having colleagues who work in educational research to connect with.

 

Scoping the research problem - There are some interesting thoughts here. The reference to Mallik and McGowan about workplace learning is potentially very helpful, although some care will be needed for drawing similarities. How feasible any research will be in my area will sometimes depend on people’s willingness to provide feedback, which is not always guaranteed as people are busy in the workplace. I will have to be careful to make use of different methodologies, as I will have to continue researching towards my dissertation.

References:

• Creswell, J. W. (2009). Research Design: qualitative, quantitative and mixed methods approaches. (3rd edition) Sage.
• Ke, F. and Hoadley, C. (2009). Evaluating Online Learning Communities. Educational Technology Research and Development, 57(4), pp.487-510.
• Newby, P. (2010). Research Methods for Education. Pearson.

 

Defining the literature review

Originally posted to eBridge, 24 February 2013

 

I've been using the questions given by Newby (2010, p.217) to further define my literature review:

• Why is this an issue/problem? With the on-going economic instability, organisations are looking for more cost-effective ways of training their staff. Managers are reluctant to release employees for extensive and off-site training, so learning technologies are becoming an area of great interest for ways of training staff remotely and more efficiently. Barriers to adoption of informal learning tools and methods arise because these are not seen as being immediately productive. However, they will become increasingly important for successful cooperation in the era of networked business (Jarche, 2013), and they do not always appear spontaneously, because the existing pressures prevent them from being realised as beneficial. Research to outline benefits of using technologies that support social learning could help to provide a much-needed theoretical basis for convincing stakeholders to adopt them, along with the benefit of self-education for practitioners.

• Is there a political dimension? Potentially, this could have a political dimension. Recent events (Garner, 2011; Orr, 2011) have brought intense political interest in the operations of exam boards. Whilst anti-bribery compliance training has been designed, developed and implemented in-house, this is subject to critical approval and auditing by government and regulatory authorities. My personal interest is in maintaining freedom in how to implement training, and ensure that it is truly successful. Jennings (2012) has questioned the general effectiveness of compliance training, as it focuses largely on 'checking boxes' and doesn't really affect behaviour in the long term. Online communities have the potential to radically alter the way in which training takes place, and ensure that real behavioural changes are achieved. Providing research data on how communities can be of benefit will help to ensure that such initiatives are not dismissed as either ineffective or unnecessary.

• When did this issue first arise? Online communities have been a point of interest for me as a practitioner since beginning my own formal learning using these methods. The concept of communities of practice has been periodically mentioned amongst colleagues, and there is a wider interest in engaging with customers using social media platforms. Online communities could provide an important practice ground for this.

• Who has an interest in the topic? All learning & development (or training) departments have an interest in the evolution of learning technologies, and the alternative methods for achieving learning outcomes that become feasible. See for instance Epic's Social Learning Debate, with interest from many high profile names in the field.

My initial thought is that this literature review should be conducted in a qualitative manner, to allow it to grow inductively. Using the principles outline by Ke & Hoadley (2009) should allow for systematic design along with a self-education aspect.

 

References:

Garner, R. (2011). Profit motive has created corrupt education system, say teachers. The Independent [online] 9 December. Available at: <http://www.independent.co.uk/news/education/education-news/profit-motive-has-created-corrupt-education-system-say-teachers-6274531.html> [Accessed 5 August 2012]

Jarche, H. [2013]. Perpetual Beta is the new reality. Life in Perpetual Beta [blog] 1 March. Available at: http://www.jarche.com/2013/03/perpetual-beta-is-the-new-reality/ [Accessed March 2013]

Jennings, C. (2012). Compliance Training: does it really work? Performance. Learning. Productivity. [blog] 15 August. Available at: <http://charles-jennings.blogspot.co.uk/2012/08/compliance-training-does-it-really-work.html> [Accessed 18 December 2012]

Ke, F. and Hoadley, C. (2009). Evaluating Online Learning Communities. Educational Technology Research and Development, 57(4), pp.487-510.

Newby, P. (2010). Research Methods for Education. Pearson Education Limited.

Orr, J. (2011). Exam boards: examiners suspended in 'corrupt practices' row. The Telegraph (online) 8 December. Available at: <http://www.telegraph.co.uk/education/secondaryeducation/8943300/Exam-boards-examiners-suspended-in-corrupt-practices-row.html> [Accessed 5 August 2012]

Blog post 5.2: Mechanism of my mind

Originally posted to eBridge, 2 March 2013

 

I've finished off my reading & reflections from last week at long last! My primary venture into reading about online community literature led me to the article mentioned in my last blog post (Ke & Hoadley, 2009). I note that the authors don't mention Salmon's (2003) model of online interaction, which has formed the basis for much of my understanding in this area, being the first model that I was introduced to on the course. It's interesting how the first mental representation we get will dominate our thinking on a subject!

As I started to outline in my previous post, the authors have done a great job of outlining how they reviewed the literature, and actually relates very well to the principles that have been introduced in our formal reading so far. Therefore I think I'll find it more helpful to pursue an approach influenced by their taxonomy, as it will hopefully allow me to connect my work with a wider body of research. However I will make a point of re-reading my work from the initial module and consider how this work affects my understanding, wither confirming it from a different angle, modifying it or destroying an incorrect assumption.

References:

• Ke, F. and Hoadley, C. (2009). Evaluating Online Learning Communities. Educational Technology Research and Development, 57(4), pp.487-510.
• Salmon, G. (2003). E-Moderating: The Key to Teaching and Learning Online. 2nd ed. London: Routledge-Falmer.

Blog post 5.1: Reviewing the literature (running behind)

Originally posted to eBridge, 24 February 2013

I'm running late this week, but thought it was better to put up a partial blog post than nothing at all. The primary piece of literature I've identified in my search is a comprehensive review of literature on online communities (Ke & Hoadley, 2009). My immediate thought is that this should save me a great deal of time, but there is a real danger of not being sufficiently cautious about accepting their methods and conclusions. The researchers do clearly set out their review, with criteria for how they chose studies that were suitable for inclusion. They also helpfully set out questions to ask of any study of online learning communities:

• What was the purpose of the study?
• What evaluation approaches were used in the study?
• What indicators or measures of online learning communities were observed in the study?
• How did the study collect and analyse data?

These will be very helpful for considering how to go about collecting my own research data. However I do need to beware that this review is made within the academic world, and likely does not include any kind of workplace learning. Participants in the online communities in question do so out of necessity - they must show that they have learned in order to progress in their studies or receive qualifications, whereas by contrast a workplace learning initiative cannot always count upon such participation from learners.

This actually seems like a good insight for how to move towards greater participation in online communities in the workplace. Having participation in online activities be part of the assessment could dramatically improve participation and uptake of learning, if used wisely, and with stakeholder support. In order to do this, I would have to show that online interactions can add value. Whilst it is unlikely a full collaborative community could be created in the first instance, progressively achieving enhancements to traditional training by provision of online communication tools could be a first step.

References:

• Ke, F. and Hoadley, C. (2009). Evaluating Online Learning Communities. Educational Technology Research and Development, 57(4), pp.487-510.