Showing posts with label Research Methods. Show all posts
Showing posts with label Research Methods. Show all posts

Sunday, 4 May 2014

Method in the madness?

Image: freeimages
So I've got my research plan ready, and now just need some final feedback from my supervisor and an ethics check for the sound research plan that I've put together. It is sound - I'm sure, aren't I? Having spent a long time reading and re-reading books on research methods, constructing, de-constructing and re-constructing my world view, I should be able to put together a set of research questions that covers all bases for both my own project and the needs of my company. Having said that, I can't help but wonder if I've caught all the main areas; if I haven't left out some critical parts under the time pressure to get a final set of questions 'out there' in time for the project going live.

So how am I going about this? I've been balancing the needs for my own learning by phrasing many of the questions in an open-ended manner that should draw on individual experiences, along with some questions aimed at evaluation of how the examiners actually use the resources. I'll also be inferring some evaluation from the kind of responses that examiners give to the open-ended questions. However there does seem to be a need for more concerted evaluation now that I look back over the questions, so I'm considering how I might go about drawing out some extra data without imposing too much on busy people!

Wednesday, 12 February 2014

Re-affirming the project outline

My project outline has remained a little vague, so I thought I would try to add more details to it...

We currently provide secondary qualifications for 14 - 19 year olds. This work is dependent on a network of some 35,000 teachers and other experts to help set and mark examinations, with technology playing an increasingly pivotal role to ensure fast and scalable transfer of marks. We are continually exploring new technologies for mark capture and transfer to ensure the best possible service for candidates.

Trialling and adopting a new marking technology requires training provision for large numbers of examiners. Previous technology adoptions have initially been dependent on government funding for their initial success, but this funding is no longer available. Training provision has increasingly moved towards the creation of online software demonstration videos and interactive simulations, now hosted on a secure Learning Management System (LMS).

We are currently piloting a new marking technology with a very small number of examiners who have received face-to-face training, and are now looking to move to exclusively online training as soon as possible. With previous technology adoptions the online provision has been developed largely through internal discussion after face-to-face training and released without a live test for examiners. For this project, the online learning materials are being developed alongside the first live pilot of the technology, with an opportunity for early testing and feedback.

We will be using action research methods to inform improvements to the online learning materials and identify additional support methods prior to general release. This research will be expanded for the live use of the software during the summer examination series, with a view to providing both evaluation of success and action research for practitioner development.

Sunday, 19 January 2014

Five questions

Image: freeimages

I'm currently working through some questions for framing qualitative research (Mason, 2002), which I hope will help me to better frame my research.

The social 'reality': Your Ontological Perspective

What is the nature of the phenomena, or entities, or 'social reality' that I wish to investigate?

Elements from Table 1.1 that appeal:
  • People, social actors, motivations, identities, cultural or social constructions
  • Experiences, development, behaviours, interactions, social processes
  • Institutions, markets, societies, organisation, connectedness, multiple realities or versions + tribes, networks
These are more relevant to my overall world-view, so I will generate an additional set specific to the current project:
  • People, understandings, perceptions,attitudes, thought
  • Experiences, development, actions, interactions, situations, rules
  • The 'material', groups, organisation

Knowledge and Evidence: Your Epistemological Position

What might represent knowledge or evidence of the entities or social 'reality' that I wish to investigate?

Experiences of users will allow me to link my design approach to how people perceive the materials and support framework - affects my ability to rationally design or re-design materials and approaches in response to user feedback.

The research would benefit from some objective measures of performance to show that knowledge about systems and procedures has successfully transferred, rather than asking for user ratings - allows for genuine accountability in our training approach.

Your Broad Research Area

What topic, or broad substantive area, is the research concerned with?

This research topic is concerned with understanding what experiences examiners have when using online resources and support to adopt new software for marking and standardisation, and any different procedures that must be adopted. My focus is on interpreting results in such a way as to allow continuous improvement of the design of such materials, and to understand how much effect the strategy has on learning, as evidenced through performance. I do not have control over the precise methods for measuring performance, but am able to access such data.

Your Intellectual Puzzle and Your Research Questions

What is the intellectual puzzle?

How do I show that my learning intervention has had the desired impact, and how can I rationally design better approaches in future, or re-design to off-set any shortcomings?

What do I wish to explain or explore?

I wish to explain how the support approaches link to performance and attitudes among the examiners, and explore how to improve both in the future.

What type of puzzle is it?

This is a causal / predictive puzzle.

Your Research Questions

What are my research questions?

For gaining insights from examiners taking part in the live pilot I'm thinking along the lines of:
  • Describe your experiences of using the software to carry out standardisation of marking
  • Describe your experiences of using the software to record and submit your marks
  • Describe your experiences of communicating with your senior examiner during the marking period
For improving on my online materials:
  • Describe your experience of using the online resources to prepare you for marking
  • Describe your experience of using the online resources and printable materials during the marking period
More of these to come - from previous experience I find that it's best to mull over these and look at what I've written down with a fresh perspective later.

Your Aims and Purpose

What is the purpose of my research? What am I doing it for?

The purpose of this research is to ensure the timely and accurate delivery of high quality marking for candidates sitting national examinations. The research will be done on behalf of my employer, to benefit examiners whilst delivering the marking and help to maintain motivation through consideration of the support offered. By achieving these objectives it is also expected that examiner performance will improve, which will be judged through the assessments from senior examiners and staff. The timeliness and accuracy of results could also be interrogated relative to expected deadlines and the number of examiners stopped or re-marked.

References

  • Mason, J., 2002. Qualitative researching, London: SAGE Publications.

Sunday, 5 January 2014

Re-examining my worldview

Image: ariss via freeimages

Research Philosophy

How do I identify with four of the main worldview areas, as discussed by Newby?

Scientism and Positivism

This school of thought demands predictable cause and effect - not easy to establish where people are concerned. Once people realise that others are able to predict their behaviour in any way, they tend to change their behaviour to avoid manipulation! It may be beneficial to look for patterns if you have a critical interest, such as a business issue, although this may actually overlap with more humanistic traditions (see below) because you will be seeking a model that is 'fit for purpose' rather than trying to establish an objective truth.

This is the case for the research carried out by Meadows (2004) and Tremain (2011, 2012), where the critical factors affecting examiner retention and job satisfaction needed to be identified. Establishing an all-encompassing theory or truth (positivism) may be too ambitious - and would be clouded by personal interests - identifying and monitoring the relevant factors was achieved through careful selection of questions and interpretation of data.


Future reading: Discussion of the work of Karl Popper (Newby, Ch.3)

Humanism, Phenomenology and Existentialism


Whist positivism demands an objective external truth independent of human influence, humanism treats 'truth' as a social construction, where one culture's truth may not be another's. This ties well with my understanding of social evolution (Hobson, 2012; Ronfeldt, 1996, 2012a,b) whereby societies develop distinct cultures of varying complexity through the addition of different forms of organisation. Conflicts arise between or even within societies due to disagreements about how society should be ordered (or not) by hierarchical institutions and free market policies. Ronfeldt (2013) acknowledges that even the term 'tribe' in his T-I-M-N framework is frequently contested by others, and progresses the model by engaging with and incorporating these disagreements into his model.

Phenomenology focuses on individual and collective experiences to form a rational basis for future action, by probing the differences between 'perceived' and 'experienced' world. Methods include description observation, reporting and reflection.


Existentialism centres around seeking to understand the world from a personal perspective, driven by conviction and desire. Methods could extend to asking participants for other forms of 'data' such as pictures, videos and stories of their lives - anything that conveys their viewpoint in richer detail than a questionnaire could. Existentialism has been applied to curriculum design, by designing a curriculum that centres around self-discovery.


Critical theory

This theory is concerned with political beliefs, particularly those that are left-of-centre and seeking to change society by making people aware of their circumstances in order to liberate them. Research carried out within a critical theory framework seeks to observe and expose individual facts which can then be combined to form an argument for change.


Aside: I am particularly interested in the references to Karl Marx, as this led to a model for social evolution around the struggle for control of wealth - somewhat akin to Ronfeldt's description of the transition between biform (T+I) and triform (T+I+M) societies.


Applying critical theory to education has been used by academics to attack the present model for education as being primarily based around servicing the needs of the capitalist economy, and thus serves only to reinforce the inequalities that it inevitably generates. However positivists would undoubtedly attack this approach because it proposes a hypothesis and seeks to prove it.

Postmodernism

This approach distinguishes itself from the 'modernist' approaches by rejecting the 'modernist' assumption that there is a single explanation for things, which leads to a natural order. For education, this almost rejects the need for theories of education, since such theories are rooted in modernist approaches! This approach does overlap with my TIMN world-view, because it is by nature multi-layered, and acknowledges that 'people and organisations can play several and sometimes conflicting roles', in keeping with the different types of organisations and the interfaces between them.


Postmodernism bears a great deal of similarity to the network (+N) principles described by Ronfeldt, with researchers seeking to act more as nodes in a network, unpicking their assumptions and sharing data that reflects their local situation, with the ideal that the value of the network increases not simply with the number of nodes but the number of connections between them.


Future reading: Steven Johnson: Future Perfect 

Summary


My adoption to the T-I-M-N framework in my outlook leads me to cross several of the boundaries of research philosophy, but perhaps I identify more strongly than I realised with the post-modernist approach for rejecting the established order with its demanded polarisation of political outlooks.

References:


Friday, 27 December 2013

The 'Carry On' factor

Image: freeimages

Following on from my thoughts about quantitative research, I'm looking at some of the dependent variables that will come into play, and thinking about how I might go about analysing them.

Intention to continue examining & job satisfaction


This is an extremely important factor for exam boards, as they are dependent on a large network of examiners to make our examinations possible. Meadows (2004) identified four factors that affect examiners' attitudes towards their jobs:
  1. The pressure and stress of examining
  2. Insight gained from examining
  3. Support from awarding body and senior examiners
  4. Pay
However, Meadows found that only the pressure and stress of examining, and the level of support received, predicted intention to continue examining; however pay did affect examiners' job satisfaction. One of the key sources of stress came from balancing examining duties with regular work, with the report recommending that resources should be diverted to lobbying for examiners to be given more time away from teaching to examine, in order to improve retention. Improving the level of support was also a recommendation, although the report notes that this would be less cost effective, since most examiners were already relatively satisfied with the support they received. Increasing pay would improve job satisfaction, but the report states that this would not improve retention.

The introduction of software tools


Tremain (2011) followed up this work to consider how the situation had changed after the introduction of electronic marking and online standardisation. The study looked at the factors that influence the satisfaction that examiners express about their work, and highlighted three factors underpinning examiners' intention to continue:

  1. The relationship between examining work and work outside examining
  2. The pressures of examining and support received
  3. The incentives for examining
The study states that although there is no imminent threat to examiner retention, future threats include the increasing use of online tools, which can contribute to examiners feeling unsupported or undervalued. Job satisfaction is considered to be more important in retention than reward for the majority of jobs, with social interaction and appropriate challenge being considered particularly valuable. The adoption of online tools had contributed to a sense of isolation amongst examiners, and also made the work more routine - although the reliability of marking has actually increased as a result.

A further study (Tremain, 2012) also set out to evaluate how specific factors involved in online marking & standardisation contributed to examiner satisfaction. This concluded that there was no significant difference in intention to continue marking between examiners who were standardised using face-to-face or online methods. Examiners who had marked using a mixture of paper and online methods showed a very slight increase in intention to continue examining. However, it was noted that the results were confounded by the different subjects and levels of experience amongst the participants.


Variables that we may be able to influence, and how:

  • Support received. By considering the different levels of support that are currently offered from the contextual model for learning (Shepherd, 2011) and identifying possible gaps, we may be able to improve the support offering for examiners in a rational way. I have already laid out some initial thoughts for this approach.
  • Insight gained from examining. Making key insights from senior examiners available in a digital form which can be shared more easily online, for instance through learning management systems or webinars, could help to ensure efficient dissemination of relevant information.
  • Social interaction. This is a long term goal that our organisation may want to consider for retaining examiners. Although we are increasingly unable to provide opportunities for examiners to meet in a face-to-face setting, there are possibilities for facilitating some more informal interaction around scheduled events. One of my colleagues is keen to run webinars for examiners to gain insight from senior examiners, and careful use of online chat could help to provide a better sense of community.
Any or all of these methods could be attempted, with measurement of the effect on intention to continue, and also examiner performance, being undertaken to determine effectiveness. One concern I have is that apparent failure to make a difference at first might result in a loss of enthusiasm for innovation, hence there would need to be trust established with stakeholders for future improvements. Undertaking action research alongside quantitative measurements to demonstrate a rational approach would be key to successful establishment of such trust.

References:

Saturday, 14 December 2013

Time for a numbers game?

Image: freeimages

Following on from my last blog post, I'm re-treading the sequence of reading from our Research Methods module to get my bearings again, and I'm coming back to the question of qualitative vs quantitative research. While I strongly identified with the Action Research methodology on my last project, it's worth deliberately opening up my mind to new possibilities, especially as there will be strong interest in some kind of numerical data from colleagues and external auditors if we are questioned on our approach.

So before I start to choose which quantitative disciplines I might wish to draw on, I'll look at the key aspects of quantitative research, consider those that appeal to me, and those I wish to avoid.



Concern with theory

Relating my findings to theory will be helpful to ensure some kind of tethers to related work, but there's a danger of getting obsessed with reproducibility and control here. When you're moving into the realm of on-demand learning, you can't guarantee learning outcomes, nor indeed that learners will even access the materials or activities that you produce for them. Newby (2010, p.96) acknowledge the limitations for educational researchers trying to identify pattern and control influences, as they are only able to view a small part of the overall education system. I would prefer to think in terms of Praxis (Wheeler, 2013), which requires practitioners to consider how closely their practice overlaps with the theories they identify with.

Concern with proof


Here lies one of the real problems for educational research - although I understand that establishing proof would give greater peace of mind, the complexity and ambiguity of the situation makes this extremely difficult:
  • The situation I face will not be the same as another practitioner does, even if our verbal descriptions of it seem similar to the untrained eye
  • The next situation that I (and the learners) face will not be the same as this one, even if it's 'just another e-marking system'
  • The time needed to establish proof would be completely at odds with the time pressures for the project, where the learning is 'on-demand'.
The best that I can hope for is to show that using theories to guide my design leads to a dependable business outcome, and that particular methods or techniques are better suited to my situation.

Identification of variables

This is one of the key aspects of quantitative theory that I see as helpful. Although my control over most independent variables involved will be limited to say the least, it will definitely be helpful to at least make some systematic efforts to identify variables in the design of materials that may be having an effect, and to measure any dependent variables which are of interest. Our particular concerns would be the performance of examiners, and intention to continue based on their experiences. Attempting to correlate these with participation in the different aspects of the support might yield useful insights into which components have succeeded, but this would have to be linked to effective practice in design.

Simply saying that an approach should be abandoned because it doesn't seem to have an effect in this situation would be potentially misleading without some understanding as to why. Creswell (2009, p.49) refers to confounding variables (e.g. discriminatory attitudes) that can come into play, which I have had some experience of when trying to introduce online learning methods in the past. Participants who are negative about the use of the tools go to great lengths to discredit them when given the opportunity to do so, whilst the majority of participants actually acknowledge a positive effect.

Conclusion

This project will benefit from the use of some quantitative approaches to analysing data about examiner performance and intention to continue, but these will need to be paired effectively with qualitative methods to understand what dependent variables relating to the choice and design of approaches might be influencing the outcomes. My next blog post will focus on the type(s) of quantitative research methods might be useful, followed by a look at rational design approaches for the learning provision.

References:

  • Creswell, J. W. (2009). Research Design: qualitative, quantitative and mixed methods approaches. (3rd edition) Sage.
  • Newby, P. (2010). Research Methods for Education. Pearson Education Limited.
  • Wheeler, S. (2013). Praxis makes perfect. Learning with e's [blog] 31 October. Available at: <http://steve-wheeler.blogspot.co.uk/2013/10/praxis-makes-perfect.html>

Tuesday, 14 May 2013

Selection of research method

Draft section of my reflective assignment, based on the blog post ‘A little more action (research) please!

I have been forced to challenge my own pre-disposition towards quantitative research methods, which was influenced by my physical sciences background. I identified this bias at an early stage in my writing, and found the distinction with qualitative research better defined in my mind by reading the comparisons made by Creswell (2009, Ch.7) and Newby (2010, Ch.3). I also realised that my previous experiences lacked any real involvement in the formal planning of research; my previous projects has always been funded without me having to submit research proposals myself. I decided that qualitative approaches seemed better suited to my context, but it took some time to fully challenge my unconscious habits. I was able to identify possible sources of bias towards theoretical models that I had used (Salmon, 2004).

However it took some additional reading (and re-reading) to fully isolate my unconscious assumptions. After extensive reading about how to create both quantitative and qualitative research proposals, I believed that I had created a set of reasonable questions for qualitative research. My initial research questions were phrased as ‘What effect does...’ and ‘How does...affect...’ Only by revisiting some of the initial reading did I notice that, despite my initial conclusions that qualitative research would be the best approach, that I had automatically designed my questions in a directive way, that would lead to bias towards theory rather than interpreting participants responses from a neutral standpoint.

After some more reading, I was able to present a much more complete and reasoned overview of my research proposal, showing a great deal more thought. This revised research plan fitted much more closely with the principles of action research (Creswell, 2009; Newby, 2010 pp.623-4; O’Brien, 1998), and represents a successful change in my thinking about research.

‘Re-learning means to abolish some toxic assumptions’ (Leonhard, 2013)

References:
  • Creswell, J. W. (2009). Research Design: qualitative, quantitative and mixed methods approaches. (3rd edition) Sage.
  • Leonhard, G. (2013). Beyond the obvious: re-defining the meaning of learning in a networked society. (video online) Available at: https://www.annotag.tv/learningtechnologies/play/18320
  • Newby, P. (2010). Research Methods for Education. Pearson Education Limited.
  • O'Brien, R. (1998). An Overview of the Methodological Approach of Action Research. (online) Available at: http://www.web.ca/~robrien/papers/xx%20ar%20final.htm (Accessed March 2013)
  • Salmon, G. (2004). e-Moderating: The Key to Teaching and Learning Online. 2nd ed. London: Routledge-Falmer.

Monday, 13 May 2013

Reliability

Is 'reliability' the right word to use for qualitative research? Cohen et al. (2007, Ch.10) suggest a variety of different words that should be used to distinguish the variable-controlled nature of quantitative research from the variable-embracing nature of qualitative research. Points addressing reliability that are present in my research include repeating the method of data collection and analysis across several groups (related to triangulation, below) and using anonymous questionnaires to encourage honesty amongst participants (p.209).

Early on in my research plan I discussed some of the principles laid out by Newby (2010, pp.120-3) for ensuring some degree of reliability in qualitative research, particularly the concept of triangulation. In the case of my project, the point about getting information form multiple sources is the best source of reliability. The questionnaire has been answered by participants from different cohorts of the same program, so we can gain some idea of how reliable (or not) the results are by comparing different cohort responses. Although the research is qualitative in nature, the reliability can be commented on by comparing the number of responses with the actual number of participants for each cohort.

References
  • Cohen, L., Manion, L., Morrison, K. & Bell, R. (2007). Research methods in education. Routledge Limited.
  • Newby, P. (2010). Research Methods for Education. Pearson Education Limited.

Sunday, 12 May 2013

Data analysis

Although I set out to do a qualitative research project, I did reserve the right to quantify my results to some extent. I've found it helpful when making sense of my data to do a quick finger-test of whether comments were positive, neutral or negative, and summarise them by question and group. These can then be totalled and presented graphically, which will be very helpful when sharing back with participants. The idea of sharing with participants as a first audience seems particularly prudent to avoid accusations of 'intellectual imperialism' (Newby, 2010, p.48). This will also act as an important feedback exercise to help practitioners see the reaction of participants, and to give me more direct contact with both parties to better understand how to inductively expand the research.

There is still a lot more work to do on full data analysis at this stage, but deadlines are looming!

References:
  • Newby, P. (2010). Research Methods for Education. Pearson Education Limited.

Collecting the data

I've been a bit lax in following the last stages of the pathway, so I thought I had better get some thoughts down before they fade away. 

What challenges do you face and what successes do you have?

Challenges:
  • Some of the experiences I am asking people to recount are in some cases from over 2 years ago.
  • The return rate for the questionnaire has been around 50% of all possible participants - partly this is due to it being a busy time of year for staff.
Successes:
  • I have received a good number of responses, with an excellent level of detail
  • Responses have yielded insights into how to improve the processes in question

What unexpected experiences do you have?

The use of an online questionnaire has left me feeling somewhat disembodied - most of the methods described in our reading concerned face-to-face methods. However they have been very effective, and in keeping with the idea that online learning should be anytime, anywhere...

How does your experience of data collection reflect, or conflict with, the principles and theories encountered in your reading?

I realise from reading through Newby (2010, pp.332-3) that I didn't pilot the questionnaire as rigorously as I should have done.  My testing was limited to sending the link to one participant who I asked about trialling it beforehand. Fortunately I had made significant revisions to my questions early on in the project that kept the research on track.

References:
  • Newby, P. (2010). Research Methods for Education. Pearson Education Limited.

Saturday, 11 May 2013

Validity

There is a danger of intertwining the concepts of reliability and validity, so I'm attempting to address them in separate posts.  Cohen, Manion & Morrison (2013, pp. 177-99) devote a great deal of attention to the concept of validity, drawing on a variety of sources to lend weight and richness to the discussion.  Several of the points (p.180) raised are applicable to my research data:
  • The natural setting is the principle source of data
  • Context-boundedness and 'thick description'
  • Data are socially situated
  • The researcher is part of the researched world
  • The researcher - rather than the research tool - is the key instrument of research
  • The data are descriptive
  • There is a concern for processes rather than simply with outcomes (inherent in my choice of action research)
  • Data are analysed inductively rather than using a priori categories
  • Data are presented in terms of the respondents rather than researchers
There are some points to beware of that might undermine the validity of my research - I will also list how these can be addressed:
  • Reactivity (internal validity) - I am aiming to improve the processes involved in the learning programs; this should not affect the data for previous cohorts, although the current cohort could potentially be affected by knowing that they are being observed.  This also touches on the issue of Researcher bias mentioned by the authors
  • Concensual validity (external) - since I am undertaking this research as a learning experience, will 'competent others' dismiss my findings due to my inexperience, or because they think I'm simply forcing the data to fit so that I pass my assessment?
Triangulation is dealt with as a means of ensuring validity.  Newby (2010, pp. 121-3) also mentions this technique, although in somewhat less detail.  So to what extent do my research methods yield themselves to this?
  • Time triangulation - I have used the same method for a number of cohorts, although this effect may be diminished because I did not carry out the survey for each group immediately following their participation, so earlier groups may not recall their experiences as accurately.
  • Theoretical triangulation - since I have avoided basing my questions on one particular theory, there is the opportunity to compare the results from the point of view of competing theories for social and online learning.
  • Investigator triangulation - the data are recorded electronically, so potentially other researchers could give their own interpretations.
  • Methodological triangulation - the same method has been used on different groups, so I can easily compare the results of each group to consider how well the results support conclusions for each group.
General points that have contributed to validity on this project:
  • Choosing an appropriate methodology for answering the research question - action research.  This has ensured that the focus is on processes - not outcomes, which I might be interested in unfairly interpreting!
  • Selecting appropriate instrumentation - using online questionnaires allowed the data to be gathered according to the time needs of respondents, and allowing them access to reminders (their forum postings) whilst responding to questions.
Limitations here include:
  • Sampling - by not requiring participation in the survey, I sacrificed control over sample sizes, which could potentially limit the validity (and reliability) of results.  However I considered that sensitivity to participants' wishes was of greater importance in this case, since some people are uncomfortable with the use of online forums in the first place.
Overall I believe I have sufficiently addressed issues of validity, but there are clearly many others that I have not encountered yet, including the points concerning data interpretation.

References
  • Cohen, L., Manion, L. & Morrison, K. (2007) Research Methods in Education.
  • Newby, P. (2010). Research Methods for Education. Pearson Education Limited.

Friday, 10 May 2013

Revisiting ethics

Some more thoughts on the ethics of my project, based on the principles outline by Newby (2010, pp.47-9)

Moral position - None of the groups involved in the study will be treated differently, since the actual participation in learning programs has already taken place. However the responses will help to guide myself and my colleagues to ensure that the use of forums is not a detriment to future learning experiences, and ensure that problems experienced by individuals can be addressed.

Honesty - I am not aiming to fit the data to a particular theory or framework from the outset, although I will have to be careful to avoid the temptation to do this. In order to provide real justification for this, such results would have to be reliable and reproducible across several data sets, which is not likely to be possible across such small samples. By choosing action research I am committing to developing understanding of the online environment for practitioners & future participants alike.

Responsibility - All information shared is within the same organisation, and responses are anonymous, with the option to avoid being quoted in written reports, so nobody should be disadvantaged as a result of this study. There are no questions that I would expect to reveal anything sensitive, however this is something to bear in mind for future studies.

Sharing - I have expressed my intent to share results with participants, and with the wider organisation. What I will need to consider is the appropriate format to write up my results for others to appreciate. I'm sure most colleagues won't get much benefit from an academic-style paper! Nor should I be disseminating detailed results outside the organisation, although there is probably some benefit in talking about the overall process on a wider platform - this blog to be precise!

Freedom - Participants have had freedom to answer the questionnaire or not, and have also been given the option to have their responses omitted from written reports. I have not based the study on their actual forum entries, and I would be inclined to avoid doing so at such an early stage in the use of forums. People are apprehensive enough about engaging with the medium without the idea of their every word being used in evidence!

Community - This research project has been discussed with colleagues in Learning & Development to ensure that the nature of questions asked is appropriate, and that participants can benefit from participating.

Attribution - This research project is not making use of anyone else's work (but then I would say that, wouldn't I?); I will however be referencing authors whose work has influenced my approach.

Standards - As mentioned previously, there are certain issues to consider about potential audiences for sharing. Whilst this blog, my research plan and my reflections are written for an academic audience, I will need to present my results in such a way that non-academic colleagues can benefit from the results. To this end I will be looking for visual ways to present the overall trends of my findings, and different ways of explaining what I believe to be the most relevant theoretical background.

Breaking down the different issues involved in ethics this way has been far more useful to me than having a code of practice laid out in any format. These questions have to be asked time and time again, for every context, to keep them from being ignored. Hopefully they won't just stay on the pages of my Wiki for assessment purposes...

References:
  • Newby, P. (2010). Research Methods for Education. Pearson Education Limited.

Sunday, 28 April 2013

A little more action (research) please!

I've been thinking about some of my learning experiences during the course, and I've picked out one of the most critical for getting my qualitative research project on the right tracks.

When I was devising my research questions, the original versions came out like this:

Primary research question:
"What effect does online interaction have on participants' approach to learning?"

Secondary research questions:

1. "How does online socialisation before a face-to-face training event affect interactions within that event?"
2. "How does online interaction before a face-to-face training event affect participants preparation for that event?"
3. "How does the opportunity for online interaction after a training event affect the application of that training in the workplace?"
4. "What would be the effect of allowing participants to contribute anonymous comments to online discourse?"

Fortunately I was reading through the advice given by Creswell (2009, Ch.7) on devising research questions for qualitative research, and I realised that my choice of words was completely inappropriate. Using the word 'affect' (or 'effect') naturally leads towards a more quantitative result because it is inherently directional for responses, as opposed to the exploratory nature of asking people to describe their experiences.

A secondary learning experience that is occurring even as I write this is revision of some of the points I had originally put into my wiki pages. I described my change in questions as being due to advice that Creswell gives specifically about action research, but I've actually misread something again - the advice was simply about qualitative versus quantitative. But I digress...

With a much clearer mind, I can re-define my overall goal for this research project as being to draw out participants' experiences of using online discussion alongside a face-to-face training event, without seeking to establish whether the effects are positive or negative through the questions themselves. So my revised set of research questions comes out as:

Primary research question:
"How would participants describe their experiences of using online interactions to support a face-to-face training event?"

Secondary research questions:
1. "Describe your experience of socialising with other participants who you interacted with online before the training event"
2. "Describe your experience of preparing for a face-to-face event where online interaction was required, relative to an event with no prior interaction"
3. "Describe your experience of participating in online interactions around course related content after the face-to-face event"


I've deliberately left out the question of anonymity - not sure I want to open that can of worms right now!

Reference:
Creswell, J. W. (2009). Research Design: qualitative, quantitative and mixed methods approaches. (3rd edition) Sage.

Thursday, 18 April 2013

Thoughts on Conrad

I've been going into more details on my literature review, and considering how other authors structure their writing, particularly from the perspective of what I find useful, and how it might influence my own writing. The first article is an interpretive study by Conrad (2002).
 
Conrad makes the point that while quantitative studies can give a useful overview of the area, understanding the experiences of users is a priority for development of communities. Although my context is somewhat different, her point about learners creating their own lines of defence sounds quite telling in the light of some initial comments observed in my study. Without understanding what barriers people put up, we can’t expect to engage meaningfully with them! Likewise, her point about our research agendas being shaped by our worldview matches with my research being guided by exploration of how network effects are re-shaping society, and the subsequent effects on education systems, both technological and organisational.

Writing a section about who, what, where, etc. is a useful step for grounding the paper, and setting the boundaries of the study, particularly the limitations of what might be achieved in the first place. She also breaks down the literature review itself to define different terms, building up how she wants the reader to understand her term of online community. She then examines each of the research questions in turn, looking at general patterns in responses, followed by particularly insightful comments by individuals. This approach will probably work very well for me, as I need to spot general patterns as well as bring out individual experiences.

Reference
Conrad, D. (2002). Deep in the Hearts of Learners: Insights into the Nature of Online Community. Journal of Distance Education, 17(1), 1-19

Sunday, 31 March 2013

Reviewing the proposal

Considering points raised by Newby (2010, Ch. 2 & 14) to refine my research proposal has given me a mixture of outcomes. Some of these confirm that my basic methodology is correct, whilst some aspects might need tweaking.

Why this research?

My primary aim in this research is gaining a qualification – however I do need to relate the research to my work context, which has been a source of tension since in future I will need to contribute to projects that further divisional goals more directly.


Why am I interested in this area?
We are aiming to move towards a blended training model, with online and face-to-face components.  The use of forums to encourage interactions among participants has been running for some time, with uncertain results.


Why do I think that it is important?
The building of learning communities with increase responsibility for their own learning will become important to sustain learning across the organisation. Our current resourcing will not enable us to meet the continuously growing demand.


Will I be able to write enough to cover my task?
I have chosen this learning program because I know that there is the potential for enough research data to be gathered for a number of cohorts, and I can combine research on the participants with benefit for practitioners.


What experience do I have of this area?
I have made use of forums in my own study and introduced the idea to colleagues through an explorative example. This led to the introduction of forums in the context that I am researching, and I have been involved in an advisory and technical capacity since then.


Who else thinks this is worth exploring?

There is a slight difference of opinion amongst my colleagues as to the value of exploring this area further. Some of my colleagues however have expressed an interest in gaining insights into this area, and even those who don’t fully support it at least acknowledge their frustrations that the use of forums is problematic.


Creating the context for the proposal


Will it deliver on our personal goals?
Although it might meet the requirements of a modular assessment, it may not have a high enough ‘impact potential’ for further progression in some respects. The project should therefore be regarded as a stepping stone for future research efforts – my key learning points will be how successful the approach is in educating practitioners and giving further insights into a learning program. Once my methodology has been established I will then look for higher impact applications.

Does the topic have a political dimension?
Online education in general is an area of intense interest, along with the tools and media for achieving success. Political commentators and politicians may well have opinions on the area, but they are likely more interested in conventional learning matters at this time.


How much do we know about the area?
As stated previously, I have been involved with the use of online discussion for learning in a variety of contexts, both academic and work related. My initial literature review has uncovered some very useful information about how online communities can be investigated methodically, hence my approach to this project.


Identifying the research boundaries

Total amount of time available
In this case the time available is dictated by the time constraints of the assessment task, and my personal time for writing up my results.

What help can we expect from others?
The work does need to be primarily my own due to the assessment criteria, and the data collection should be fairly straightforward due to sending out questionnaires by e-mail. I do need to consider how much time it might take participants to answer my questionnaires, given that we will be approaching a time of higher workload.

Do we need specialist help?
No outside help will be necessary in this case since the data collection and interpretation is of my own devising, and I have sufficient access to journal articles at this time.

What if things don’t go according to plan?
I believe I have sufficient safety in terms of preserving data and the actual collection. My concern is with the possible response rate of participants within the time frame. For this reason I had already opted to research a program that has had multiple cohorts, and also the interactions have been digitally recorded due to the nature of the medium.

Assessing ethical issues

This has already been covered in the blog post on principles to some extent. As a summary, I will be carrying out my research within my workplace and sharing those results with colleagues. One potential issue could come from the sharing of participants’ responses to a wider audience, from a forum that was set up as a confidential area. For this reason I will need to ask colleagues about how to use their responses. I will treat responses anonymously by default, possibly with an option for their responses to not be quoted in reports. All postings on forums will remain confidential.

Preparing a case for a research programme

This research programme is for assessment purposes, but setting out my thinking will be of use in future, especially if I want to keep research as a positive influence on our evaluation strategy.

Getting an idea for research

From the sources of stimulus I have identified several different strands to my research idea & motivation:

1.   Literature
My review of the literature has revealed that there are a number of different theories cited by researchers, in some cases not including those that I have used as a theoretical basis for my own learning. Because of this I am wary of using quantitative methodology to tie results to a particular theory

2.   Policy
Ongoing interest in the field of online education interests me, particularly from the point of view of keeping practitioner education a key issue. Research that is carried out purely from a top-down institutional perspective is unlikely to produced sustainable benefits for the education system or society.

3.   Experience
Following on from previous points, my experience of online education is that it is most powerful as a constructivist environment, to support communities of inquiry and learning by doing.

Potential to finish research project

For this assessed research project I am confident that I can deliver the required results, and show that my methodology for design and data collection has been adequate. Questions about the impact of my research will need to be addressed for future efforts, since online learning projects will be focussed on areas of greatest business interest.

Arguing the case for the research

(a) Can the topic convince our judges?
The project should sufficiently convince my academic judges, although from a work perspective I will need to convince colleagues that the project has sufficient merit as a developmental tool for future work, and show that my approach to sharing results and insights can benefit the wider team.

(b) Is it feasible for me to undertake it?
I believe that this research project is well within my capabilities, and can be completed satisfactorily within the given time frame.


(i)  The field of enquiry
Research in this area is nothing particularly new, since there has been significant research into the field by academic researchers. Neither should this be considered as a theoretical study, although the context will always be different from others. The real interest here is how policies of using online education methods are succeeding against our objectives, and how the use of these approaches compares between academic origins and the actual implementation in the workplace.

(ii) The research issue / question
Related to the issues from previous questions, my key questions for research are based around the experiences of people participating in the use of new types of media, and ensuring that practitioner approaches can match the desire for business results.

Making sense of what we have

My interpretation of results may well draw on theory, although this will not be the primary lens. My main focus will be on judging whether activities have met objectives and how they can be modified for improvements, and also to identify what course of action should not be taken. My approach to truth here is realising how others perceive it, and using this to inform practice.

References
  • Newby, Peter (2010).  Research Methods for Education Chapter 2 p.31-65