Thinking About Me: Reflexivity in science and qualitative research

self rembrandt reflexivity

Reflexivity is a process (and it should be a continuing process) of reflecting on how the researcher could be influencing a research project.


In a traditional positivist research paradigm, the researcher attempts to be a neutral influence on  research. They make rational and logical interpretations, and assume a ‘null hypothesis’, in which they expect all experiments to have no effect, and have no pre-defined concept of what the research will show.


However, this is a lofty aspiration and difficult to achieve in practice. Humans are fallible and emotional beings, with conflicting pressures on jobs, publication records and their own hunches. There are countless stories of renowned academics having to retract papers, or their whole research careers because of faked results, flawed interpretations or biased coding procedures.


Many consider it to be impossible to fully remove the influence of the researcher from the process, and so all research would be ‘tainted’ in some way by the prejudices of those in the project. This links into the concept of “implicit bias” where even well-meaning individuals are influenced by subconscious prejudices. These have been shown to have a significant discriminatory impact on pay, treatment in hospitals and recruitment along lines of gender and ethnicity.


So does this mean that we should abandon research, and the pursuit of truly understanding the world around us? No! Although we might reject the notion of attaining an absolute truth, that doesn’t mean we can’t learn something. Instead of pretending that the researcher is an invisible and neutral piece of the puzzle, a positionality and reflexivity approach argues that the background of the researcher should be detailed in the same way as the data collection methods and analytical techniques.


But how is this done in practice? Does a researcher have to bare their soul to the world, and submit their complete tax history? Not quite, but many in feminist and post-positivist methodologies will create a ‘positionality statement’ or ‘reflexivity statement’. This is a little like a CV or self-portrait of potential experiences and bias, in which the researcher is honest about personal factors that might influence their decisions and interpretations. These might include the age, gender, ethnicity and class of the researcher, social and research issues they consider important, their country and culture, political leanings, life experiences and education. In many cases a researcher will include such a statement with their research publications and outputs, just Googling ‘positionality statements’ will provide dozens of links to examples.

 

However, I feel that this is a minimum level of engagement with the issue, and it’s actually important to keep a reflexive stance throughout the research process. Just like how a one-off interview is not as accurate a record as a daily diary, keeping reflexivity notes as an ongoing part of a research journal is much more powerful. Here a researcher can log changes in their situation, assumptions and decisions made throughout the research process that might be affected by their personal stance. It’s important that the researcher is constantly aware of when they are making decisions, because each is a potential source of influence. This includes deciding what to study, who to sample, what questions to ask, and which sections of text to code and present in findings.


Why this is especially pertinent to qualitative research? It’s often raised in social science, especially ethnography and close case study work with disadvantaged or hard-to-reach populations where researchers have a much closer engagement with their subjects and data. It could be considered that there are more opportunities for personal stance to have an impact here, and that many qualitative methods, especially the analysis process using grounded theory, are open to multiple interpretations that vary by researcher. Many make the claim that qualitative research and data analysis is more subjective than quantitative methods, but as we’ve argued above, it might be better to say that they are both subjective. Many qualitative epistemological approaches are not afraid of this subjectivity, but will argue it is better made forthright and thus challenged, rather than trying to keep it in the dark.


Now, this may sound a little crazy, especially to those in traditionally positivist fields like STEM subjects (Science, Technology Engineering, Mathematics). Here there is generally a different move: to use process and peer review to remove as many aspects of the research that are open to subjective interpretation as possible. This direction is fine too!


However, I would argue that researchers already have to make a type of reflexivity document: a conflict of interest statement. Here academics are supposed to declare any financial or personal interest in the research area that might influence their neutrality. This is just like a positionality statement! An admission that researchers can be influenced by prejudices and external factors, and that readers should be aware of such conflicts of interest when doing their own interpretation of the results.


If it can be the case that money can influence science (and it totally can) it’s also been shown that gender and other aspects of an academic's background can too. All reflexivity asks us to do is be open and honest with our readers about who we are, so they can better understand and challenge the decisions we make.

 

 

Like all our blog articles, this is intended to be a primer on some very complex issues. You’ll find a list of references and further reading below (in addition to the links included above). Don’t forget to try Quirkos for all your qualitative data analysis needs! It can help you keep, manage and code a reflexive journal throughout your analysis procedure. See this blog article for more!

 

 

References

 

Bourke, B., 2014, Positionality: Reflecting on the Research Process, The Qualitative Report 19, http://www.nova.edu/ssss/QR/QR19/bourke18.pdf


Day, E., 2002, Me, My*self and I: Personal and Professional Re-Constructions in Ethnographic Research, FQS 3(3) http://www.qualitative-research.net/index.php/fqs/article/view/824/1790


Greenwald, A., Krieger, L., 2006, Implicit Bias: Scientific Foundations, California Law Review, 94(4). http://www.jstor.org/stable/20439056


Lynch, M., 2000, Against Reflexivity as an Academic Virtue and Source of Privileged Knowledge, Theory, Culture & Society 17(3), http://tcs.sagepub.com/content/17/3/26.short


Savin-Baden, M., Major C., 2013, Personal stance, positionality and reflexivity, in Qualitative Research: The essential guide to theory and practice. Routledge, London.


Soros, G., 2013, Fallibility, reflexivity and the human uncertainty principle, Journal of Economic Methodology, 20(4) https://www.georgesoros.com/essays/fallibility-reflexivity-and-the-human-uncertainty-principle-2/

 

 

The importance of keeping open-ended qualitative responses in surveys

open-ended qualitative responses in surveys

I once had a very interesting conversation at a MRS event with a market researcher from a major media company. He told me that they were increasingly ‘costing-out’ the qualitative open-ended questions from customer surveys because they were too expensive and time consuming to analyse. Increasingly they were replacing open-ended questions with a series of Likert scale questions which could be automatically and statistically examined.

 

I hear similar arguments a lot, and I totally understand the sentiment: doing good qualitative research is expensive, and requires good interpretation. However, it’s just as possible to do statistical analysis poorly, and come up with meaningless and inaccurate answers. For example, when working with Likert scales, you have to be careful about which parametric tests you use, and make sure that the data is normally distributed (Sullivan and Artino 2013).

 

There is evidence that increasing the number of options in closed questions does not significantly change the responses participants share (Dawes 2008), so if you need a good level of nuance into customer perceptions, why not let your users choose their own words. “Quick Qual” approaches, like asking people to use one word to describe the product or their experience can be really illuminating. Better yet, these responses are easy to analyse, and present as an engaging word cloud!

 

Even when you have longer responses, it’s not necessary to always take a full classification and quantification approach to qualitative survey data such as in Nardo (2003). For most market research investigations, this level of detail is not needed by researcher or client.

 

Indeed, you don’t need to do deep analysis of the data to get some value from it. A quick read through some of the comments can make sure your questions are on track, and there aren’t other common issues being raised. It helps check you were asking the right questions, and can help explain why answers for some people aren’t matching up with the rest. As ever, qualitative data is great for surprises, responses you hadn’t thought of, and understanding motivations.

 

Removing open ended questions means you can’t provide nice quotes or verbatims from the feedback, which are great for grounding a report and making it come to life. If you have no quotes from respondents, you also are missing the opportunity to create marketing campaigns around comments from customer evangelists, something Lidl UK has done well by featuring positive Tweets about their brand. In this article marketing director Claire Farrant notes the importance of listening and engaging with customer feedback in this way. It can also make people more satisfied with the feedback process if they have a chance to voice their opinions in more depth.

 

I think it’s also vital to include open-ended questions when piloting a survey or questionnaire. Having qualitative data at an early stage can let you refine your questions, and the possible responses. Sometimes the language used by respondents is important to reflect when setting closed questions: you don’t want to be asking questions like “How practical did you find this product” when the most common term coming from the qualitative data is “Durable”. It’s not always necessary to capture and analyse qualitative data for thousands of responses, but looking at a sample of a few dozen or hundred can show if you are on the right track before a big push.

 

You also shouldn’t worry too much about open-ended surveys having lower completion rates. A huge study by SurveyMonkey found that a single open question actually increased engagement slightly, and only when there were 5 or more open-ended response boxes did this have a negative impact on completion.

 

Finally, without qualitative responses, you lose the ability to triangulate and integrate your qualitative and quantitative data: one of the most powerful tools in survey analysis. For example, in Quirkos it is trivial to do very quick comparative subset analysis, using any of the closed questions as a pivot point. So you can look at the open ended responses from people who gave high satisfaction scores next to those that were low, and rather than then being stuck trying to explain the difference in opinion, you can look at the written comments to get an insight into why they differ.

 

And I think this is key to creating good reports for clients. Usually, the end point for a customer is not being told that 83% of their customers are satisfied with their helpline: they want to actions that will improve or optimise delivery. What exactly was the reason 17% of people had a bad experience? It’s all very well to create an elaborate chain of closed questions, such as ‘You said you were unsatisfied. Which of these reasons bests explains this? You said the response time made you unsatisfied. How long did you wait? 0-3min, 3-5min etc. etc. But these types of surveys are time consuming to program and make comprehensive, and sometimes just allowing someone to type “I had to wait more than 15 minutes for a response” would have given you all the data you needed on a critical point.

 

The depth and insight from qualitative data can illuminate differences in respondent’s experiences, and give the key information to move things forward. Instead of thinking how can you cost-out qualitative responses, think instead how you can make sure they are integrated to provide maximum client value! A partnership between closed and open questions is usually the most powerful way to get both a quick summary and deep insight into complex interactions, and there is no need to be afraid of the open box!

 

Quirkos is designed to make it easy to bring both qualitative and quantitative data from surveys together, and use the intuitive visual interface to explore and play with market research data. Download a free trial of our qualitative analysis software, or contact us for a demo, and see how quickly you can step-up from paper based analysis into a streamlined and insightful MRX workflow!

 

Analytical memos and notes in qualitative data analysis and coding

Image adapted from https://commons.wikimedia.org/wiki/File:Male_forehead-01_ies.jpg - Frank Vincentz

There is a lot more to qualitative coding than just deciding which sections of text belong in which theme. It is a continuing, iterative and often subjective process, which can take weeks or even months. During this time, it’s almost essential to be recording your thoughts, reflecting on the process, and keeping yourself writing and thinking about the bigger picture. Writing doesn’t start after the analysis process, in qualitative research it often should precede, follow and run in parallel to a iterative interpretation.


The standard way to do this is either through a research journal (which is also vital during the data collection process) or through analytic memos. Memos create an important extra level of narrative: an interface between the participant’s data, the researcher’s interpretation and wider theory.


You can also use memos as part of a summary process, to articulate your interpretations of the data in a more concise format, or even throw the data wider and larger by drawing from larger theory.


It’s also a good cognitive exercise: regularly make yourself write what you are thinking, and keep yourself articulating yourself. It will make writing up at the end a lot easier in the end! Memos can be a very flexible tool, and qualitative software can help keep these notes organised. Here are 9 different ways you might use memos as part of your work-flow for qualitative data analysis:

 

Surprises and intrigue
This is probably the most obvious way to use memos: note during your reading and coding things that are especially interesting, challenging or significant in the data. It’s important to do more than just ‘tag’ these sections, reflect to yourself (and others) why these sections or statements stand out.

 

Points where you are not sure
Another common use of memos is to record sections of the data that are ambiguous, could be interpreted in different ways, or just plain don’t fit neatly in to existing codes or interpretations. But again, this should be more than just ‘flagging’ bits that need to be looked at again later, it’s important to record why the section is different: sometimes the act of having to describe the section can help comprehension and illuminate the underlying causation.

 

Discussion with other researchers
Large qualitative research projects will often have multiple people coding and analysing the data. This can help to spread the workload, but also allows for a plurality of interpretations, and peer-checking of assumptions and interpretations. Thus memos are very important in a team project, as they can be used to explain why one researcher interpreted or coded sources in a certain way, and flag up ambiguous or interesting sections for discussion.

 

Paper-trail
Even if you are not working as part of a team, it can be useful to keep memos to explain your coding and analytical choices. This may be important to your supervisors (or viva panel) as part of a research thesis, and can be seen as good practice for sharing findings in which you are transparent about your interpretations. There are also some people with a positivist/quantitative outlook who find qualitative research difficult to trust because of the large amount of seemingly subjective interpretation. Memos which detail your decision making process can help ‘show your working out’ and justify your choices to others.

 

Challenging or confirming theory
This is another common use of memos, to discuss how the data either supports or challenges theory. It is unusual for respondents to neatly say something like “I don’t think my life fits with the classical structure of an Aeschylean tragedy” should this happen to be your theoretical approach! This means you need to make these observations and higher interpretation, and note how particular statements will influence your interpretations and conclusions. If someone says something that turns your theoretical framework on its head, note it, but also use the memos as a space to record context that might be used later to explain this outlier. Memos like this might also help you identify patterns in the data that weren’t immediately obvious.

 

Questioning and critiquing the data/sources
Respondents will not always say what they mean, and sometimes there is an unspoken agenda below the surface. Depending on the analytical approach, an important role of the researcher is often to draw deeper inferences which may be implied or hinted at by the discourse. Sometimes, participants will outright contradict themselves, or suggest answers which seem to be at odds with the rest of what they have shared. It’s also a great place to note the unsaid. You can’t code data that isn’t there, but sometimes it’s really obvious that a respondent is avoiding discussing a particular issue (or person). Memos can note this observation, and discuss why topics might be uncomfrotable or left out in the narrative.


Part of an iterative process
Most qualitative research does not follow a linear structure, it is iterative and researchers go back and re-examine the data at different stages in the process. Memos should be no different, they can be analysed themselves, and should be revisited and reviewed as you go along to show changes in thought, or wider patterns that are emerging.


Record your prejudices and assumptions
There is a lot of discussion in the literature about the importance of reflexivity in qualitative research, and recognising the influence of the non-neutral researcher voice. Too often, this does not go further than a short reflexivity/positionality statement, but should really be a constantly reconsidered part of the analytical process. Memos can be used as a prompt and record of your reflexive process, how the data is challenges your prejudices, or how you might be introducing bias in the interpretation of the data.


Personal thoughts and future directions
As you go through the data, you may be noticing interesting observations which are tangential, but might form the basis of a follow-on research project or reinterpretation of the data. Keeping memos as you go along will allow you to draw from this again and remember what excited you about the data in the first place.

 

 

Qualitative analysis software can help with the memo process, keeping them all in the same place, and allowing you to see all your memos together, or connected to the relevant section of data. However, most of the major software packages (Quirkos included) don’t exactly forefront the memo tools, so it is important to remember they are there and use them consistently through the analytical process.

 

Memos in Quirkos are best done using a separate source which you edit and write your memos in. Keeping your notes like this allows you to code your memos in the same way you would with your other data, and use the source properties to include or exclude your memos in reports and outputs as needed. However, it can be a little awkward to flip between the memo and active source, and there is currently no way to attach memos to a particular coding event. However, this is something we are working on for the next major release, and this should help researchers to keep better notes of their process as they go along. More detail on qualitative memos in Quirkos can be found in this blog post article.

 

 

There is a one-month free trial of Quirkos, and it is so simple to use that you should be able to get going just by watching one of our short intro videos, or the built-in guide. We are also here to help at any stage of your process, with advice about the best way to record your analytical memos, coding frameworks or anything else. Don’t be shy, and get in touch!

 


References and further reading:


Chapman, Y., Francis, K., 2008. Memoing in qualitative research, Journal of Research in Nursing, 13(1). http://jrn.sagepub.com/content/13/1/68.short?rss=1&ssource=mfc

 

Gibbs, G., 2002, Writing as Analysis, http://onlineqda.hud.ac.uk/Intro_QDA/writing_analysis.php

Saldana, J., 2015, The Coding Manual for Qualitative Researchers, Writing Analytic Memos about Narritative and Visual Data, Sage, London. https://books.google.co.uk/books?id=ZhxiCgAAQBAJ

 

 

Starting a qualitative research thesis, and choosing a CAQDAS package

qualitative thesis software

 

For those about to embark on a qualitative Masters or PhD thesis, we salute you!

 

More and more post-graduate students are using qualitative methods in their research projects, or adopting mixed-method data collection and using a small amount of qualitative data which needs to be combined with quantitative data. So this year, how can students decide the best approach for the analysis of their data, and can CAQDAS or QDA software help their studies?

 

First, as far as possible, don’t chose the software, choose the method. Think about what you are trying to research, the best way to get deep data to answer your research questions. The type and amount of data you have will be an important factor. Next, how much existing literature and theory there is around your research area? This will affect whether you will adopt a grounded theory approach, or will be testing or challenging existing theory.

 

Indeed, you may decide that that you don’t need software for your research project. For small projects, especially case studies, you may be more comfortable using printouts of your data, and while reading mark important sections with highlighters and post-it notes. Read Séror (2005) for a comparison of computer vs paper methods. You could also look at the 5 Level QDA, an approach to planning and learning the use of qualitative software so that you develop strategies and tactics that help you make the most of the QDA software.

 

Unfortunately, if you decide you want to use a particular software solution it’s not always as simple as it should be. You will have to eventually make a practical choice based on what software your university has, what support they provide, and what your peers and supervisors use.

 

However, while you are a student, it’s also a good time to experiment and see what works best for you. Not only do all the major qualitative software packages offer a free trial, student licences are hugely discounted against the full versions. This gives you the option to buy a copy for yourself (for a relatively small amount of money).

 

There’s a lot of variety in the different qualitative data analysis software available. The most common one is Nvivo, which your university or department may already have a licence for. This is a very powerful package, but can be intimidating for first-time users. Common alternatives like MAXQDA or Atlas.ti are more user friendly, but also adopt similar spreadsheet-like interfaces. There are also lots of more niche alternatives, for example Transana is unmatched for video analysis, and Dedoose works entirely in the cloud so you can access it from any computer. For a more comprehensive list, check out the Wikipedia list, or the profiles on textanalysis.info

 

Quirkos does a couple of things differently though. First, our student licences don’t expire, and are some of the cheapest around. This means that it doesn’t matter if your PhD takes 3 or 13 years, you will still be able to access your work and data without paying again. And yes, you can keep using your licence into your professional career. It also aims to be the easiest software package to use, and puts visualisations of the data first and foremost in the interface.

 

So give Quirkos a try, but don’t forget about all the other alternatives out there: between them all you will find something that works in the way you want it to and makes your research a little less painful!

 

 

Reflections on qualitative software from KWALON 2016

rotterdam centraal station

Last week saw a wonderful conference held by the the Dutch network for qualitative research KWALON, based at the Erasmus University, Rotterdam. The theme was no less than the future of Qualitative Data Analysis (QDA) software.

 

Chair Jeanine Evers opened the session by outlining 8 important themes the group had identified on qualitative analysis software.

 

The first was the challenge of adding features to software that is requested by users or present in competitors software, without breaking the underlying design of the software. Quirkos really connects to this theme, because we have always tried to have a very simple tool-set, based on a philosophy that the software should be very easy to use. While we obviously take heed of suggestions made by our users, we actually have a comprehensive and limited set of features which we have always planned to introduce, and will continue delivering these over the next few years.

 

However, it is not the intention of Quirkos to become a large software package with lots of features, something Jeanine described as a ‘obese software’ that needs to be put on a diet. It was noted that many software providers have released ‘lite’ versions of their software, and another discussion point was if this fragmented approach can benefit universities and users.

 

User friendliness was another theme of the session, and by keeping Quirkos simple we hope to always have this at the fore of our design philosophy. In my talk (you can now get the slides here) I discussed these themes as mostly being about improving accessibility. To this end, we have tried to make Quirkos not just easier to use, but also to teach and own, with permanent licences and discounts for researchers from  countries that can’t usually afford this type of software. For us, the long-term goal is not just increasing the number of people that use software for qualitative analysis, but the number that are able to take up qualitative research in general.

 

There was also some good discussion at the end of our talk about the risks of making software easy to use: especially that it also makes it easy to use badly. As we’ve discussed many times on this blog, software in general can make it very satisfying to code, and this can appear to be more productive than stepping back and thinking about themes or a undertaking deep readings of the data. These problems can apply to all software packages, so it is important that students and educators work together to learn about the whole analysis procedure, and what part CAQDAS can play.

 

Comments also touched on how memo making is a critical part of a good iterative and reflexive qualitative analysis process: which at the moment Quirkos doesn’t forefront (see for example how F4analyse and a future version of Cassandre will operate). Although it is possible to record memos by typing in a source, which gives you the ability to tag and code your memos, as well as writing notes as source properties, this is currently not highlighted enough and we plan on revamping the memo features in a future update.

 


The final theme of the conference, and a major push, was to promote a standard way to exchange software between qualitative software. At the moment it is very difficult for users to move their coded data from one software package to the other. Although most major packages provide options to export their data to other formats (such as spreadsheet CSV data like Quirkos), there is currently no single standard for how should be formatted, so it is very difficult to bring this data – complete with themes and coding - into another package.

 

There was strong support from the software developers to develop and support such a standard, as well as discussions about existing initiatives such as CATA-XML and QuDEx.


This is very important: but not just for users of different of qualitative analysis software, who want to be able to collaborate with universities and colleagues who use different packages. It’s also important for archival purposes, so that qualitative coded data can be universally shared and stored for secondary analysis, and to make it easier for data to be brought in for analysis from the huge number of digital sources in the digital humanities, such as history, journalism, and social media. Such a standard could also be important for formatting data so that machine learning and natural language processing can automate some of the simpler analysis processes on very large ‘big-data’ datasets.


So there is a lot to be done, but a lot of interest in the area in the next few years, with major and minor players all taking different approaches, and seeking common ground. Quirkos is honoured to be a small part of this, and will do whatever we can to improve the world of qualitative analysis for this and the next generation of researchers.