Thinking About Me: Reflexivity in science and qualitative research

self rembrandt reflexivity

Reflexivity is a process (and it should be a continuing process) of reflecting on how the researcher could be influencing a research project.


In a traditional positivist research paradigm, the researcher attempts to be a neutral influence on  research. They make rational and logical interpretations, and assume a ‘null hypothesis’, in which they expect all experiments to have no effect, and have no pre-defined concept of what the research will show.


However, this is a lofty aspiration and difficult to achieve in practice. Humans are fallible and emotional beings, with conflicting pressures on jobs, publication records and their own hunches. There are countless stories of renowned academics having to retract papers, or their whole research careers because of faked results, flawed interpretations or biased coding procedures.


Many consider it to be impossible to fully remove the influence of the researcher from the process, and so all research would be ‘tainted’ in some way by the prejudices of those in the project. This links into the concept of “implicit bias” where even well-meaning individuals are influenced by subconscious prejudices. These have been shown to have a significant discriminatory impact on pay, treatment in hospitals and recruitment along lines of gender and ethnicity.


So does this mean that we should abandon research, and the pursuit of truly understanding the world around us? No! Although we might reject the notion of attaining an absolute truth, that doesn’t mean we can’t learn something. Instead of pretending that the researcher is an invisible and neutral piece of the puzzle, a positionality and reflexivity approach argues that the background of the researcher should be detailed in the same way as the data collection methods and analytical techniques.


But how is this done in practice? Does a researcher have to bare their soul to the world, and submit their complete tax history? Not quite, but many in feminist and post-positivist methodologies will create a ‘positionality statement’ or ‘reflexivity statement’. This is a little like a CV or self-portrait of potential experiences and bias, in which the researcher is honest about personal factors that might influence their decisions and interpretations. These might include the age, gender, ethnicity and class of the researcher, social and research issues they consider important, their country and culture, political leanings, life experiences and education. In many cases a researcher will include such a statement with their research publications and outputs, just Googling ‘positionality statements’ will provide dozens of links to examples.

 

However, I feel that this is a minimum level of engagement with the issue, and it’s actually important to keep a reflexive stance throughout the research process. Just like how a one-off interview is not as accurate a record as a daily diary, keeping reflexivity notes as an ongoing part of a research journal is much more powerful. Here a researcher can log changes in their situation, assumptions and decisions made throughout the research process that might be affected by their personal stance. It’s important that the researcher is constantly aware of when they are making decisions, because each is a potential source of influence. This includes deciding what to study, who to sample, what questions to ask, and which sections of text to code and present in findings.


Why this is especially pertinent to qualitative research? It’s often raised in social science, especially ethnography and close case study work with disadvantaged or hard-to-reach populations where researchers have a much closer engagement with their subjects and data. It could be considered that there are more opportunities for personal stance to have an impact here, and that many qualitative methods, especially the analysis process using grounded theory, are open to multiple interpretations that vary by researcher. Many make the claim that qualitative research and data analysis is more subjective than quantitative methods, but as we’ve argued above, it might be better to say that they are both subjective. Many qualitative epistemological approaches are not afraid of this subjectivity, but will argue it is better made forthright and thus challenged, rather than trying to keep it in the dark.


Now, this may sound a little crazy, especially to those in traditionally positivist fields like STEM subjects (Science, Technology Engineering, Mathematics). Here there is generally a different move: to use process and peer review to remove as many aspects of the research that are open to subjective interpretation as possible. This direction is fine too!


However, I would argue that researchers already have to make a type of reflexivity document: a conflict of interest statement. Here academics are supposed to declare any financial or personal interest in the research area that might influence their neutrality. This is just like a positionality statement! An admission that researchers can be influenced by prejudices and external factors, and that readers should be aware of such conflicts of interest when doing their own interpretation of the results.


If it can be the case that money can influence science (and it totally can) it’s also been shown that gender and other aspects of an academic's background can too. All reflexivity asks us to do is be open and honest with our readers about who we are, so they can better understand and challenge the decisions we make.

 

 

Like all our blog articles, this is intended to be a primer on some very complex issues. You’ll find a list of references and further reading below (in addition to the links included above). Don’t forget to try Quirkos for all your qualitative data analysis needs! It can help you keep, manage and code a reflexive journal throughout your analysis procedure. See this blog article for more!

 

 

References

 

Bourke, B., 2014, Positionality: Reflecting on the Research Process, The Qualitative Report 19, http://www.nova.edu/ssss/QR/QR19/bourke18.pdf


Day, E., 2002, Me, My*self and I: Personal and Professional Re-Constructions in Ethnographic Research, FQS 3(3) http://www.qualitative-research.net/index.php/fqs/article/view/824/1790


Greenwald, A., Krieger, L., 2006, Implicit Bias: Scientific Foundations, California Law Review, 94(4). http://www.jstor.org/stable/20439056


Lynch, M., 2000, Against Reflexivity as an Academic Virtue and Source of Privileged Knowledge, Theory, Culture & Society 17(3), http://tcs.sagepub.com/content/17/3/26.short


Savin-Baden, M., Major C., 2013, Personal stance, positionality and reflexivity, in Qualitative Research: The essential guide to theory and practice. Routledge, London.


Soros, G., 2013, Fallibility, reflexivity and the human uncertainty principle, Journal of Economic Methodology, 20(4) https://www.georgesoros.com/essays/fallibility-reflexivity-and-the-human-uncertainty-principle-2/

 

 

The importance of keeping open-ended qualitative responses in surveys

open-ended qualitative responses in surveys

I once had a very interesting conversation at a MRS event with a market researcher from a major media company. He told me that they were increasingly ‘costing-out’ the qualitative open-ended questions from customer surveys because they were too expensive and time consuming to analyse. Increasingly they were replacing open-ended questions with a series of Likert scale questions which could be automatically and statistically examined.

 

I hear similar arguments a lot, and I totally understand the sentiment: doing good qualitative research is expensive, and requires good interpretation. However, it’s just as possible to do statistical analysis poorly, and come up with meaningless and inaccurate answers. For example, when working with Likert scales, you have to be careful about which parametric tests you use, and make sure that the data is normally distributed (Sullivan and Artino 2013).

 

There is evidence that increasing the number of options in closed questions does not significantly change the responses participants share (Dawes 2008), so if you need a good level of nuance into customer perceptions, why not let your users choose their own words. “Quick Qual” approaches, like asking people to use one word to describe the product or their experience can be really illuminating. Better yet, these responses are easy to analyse, and present as an engaging word cloud!

 

Even when you have longer responses, it’s not necessary to always take a full classification and quantification approach to qualitative survey data such as in Nardo (2003). For most market research investigations, this level of detail is not needed by researcher or client.

 

Indeed, you don’t need to do deep analysis of the data to get some value from it. A quick read through some of the comments can make sure your questions are on track, and there aren’t other common issues being raised. It helps check you were asking the right questions, and can help explain why answers for some people aren’t matching up with the rest. As ever, qualitative data is great for surprises, responses you hadn’t thought of, and understanding motivations.

 

Removing open ended questions means you can’t provide nice quotes or verbatims from the feedback, which are great for grounding a report and making it come to life. If you have no quotes from respondents, you also are missing the opportunity to create marketing campaigns around comments from customer evangelists, something Lidl UK has done well by featuring positive Tweets about their brand. In this article marketing director Claire Farrant notes the importance of listening and engaging with customer feedback in this way. It can also make people more satisfied with the feedback process if they have a chance to voice their opinions in more depth.

 

I think it’s also vital to include open-ended questions when piloting a survey or questionnaire. Having qualitative data at an early stage can let you refine your questions, and the possible responses. Sometimes the language used by respondents is important to reflect when setting closed questions: you don’t want to be asking questions like “How practical did you find this product” when the most common term coming from the qualitative data is “Durable”. It’s not always necessary to capture and analyse qualitative data for thousands of responses, but looking at a sample of a few dozen or hundred can show if you are on the right track before a big push.

 

You also shouldn’t worry too much about open-ended surveys having lower completion rates. A huge study by SurveyMonkey found that a single open question actually increased engagement slightly, and only when there were 5 or more open-ended response boxes did this have a negative impact on completion.

 

Finally, without qualitative responses, you lose the ability to triangulate and integrate your qualitative and quantitative data: one of the most powerful tools in survey analysis. For example, in Quirkos it is trivial to do very quick comparative subset analysis, using any of the closed questions as a pivot point. So you can look at the open ended responses from people who gave high satisfaction scores next to those that were low, and rather than then being stuck trying to explain the difference in opinion, you can look at the written comments to get an insight into why they differ.

 

And I think this is key to creating good reports for clients. Usually, the end point for a customer is not being told that 83% of their customers are satisfied with their helpline: they want to actions that will improve or optimise delivery. What exactly was the reason 17% of people had a bad experience? It’s all very well to create an elaborate chain of closed questions, such as ‘You said you were unsatisfied. Which of these reasons bests explains this? You said the response time made you unsatisfied. How long did you wait? 0-3min, 3-5min etc. etc. But these types of surveys are time consuming to program and make comprehensive, and sometimes just allowing someone to type “I had to wait more than 15 minutes for a response” would have given you all the data you needed on a critical point.

 

The depth and insight from qualitative data can illuminate differences in respondent’s experiences, and give the key information to move things forward. Instead of thinking how can you cost-out qualitative responses, think instead how you can make sure they are integrated to provide maximum client value! A partnership between closed and open questions is usually the most powerful way to get both a quick summary and deep insight into complex interactions, and there is no need to be afraid of the open box!

 

Quirkos is designed to make it easy to bring both qualitative and quantitative data from surveys together, and use the intuitive visual interface to explore and play with market research data. Download a free trial of our qualitative analysis software, or contact us for a demo, and see how quickly you can step-up from paper based analysis into a streamlined and insightful MRX workflow!

 

Analytical memos and notes in qualitative data analysis and coding

Image adapted from https://commons.wikimedia.org/wiki/File:Male_forehead-01_ies.jpg - Frank Vincentz

There is a lot more to qualitative coding than just deciding which sections of text belong in which theme. It is a continuing, iterative and often subjective process, which can take weeks or even months. During this time, it’s almost essential to be recording your thoughts, reflecting on the process, and keeping yourself writing and thinking about the bigger picture. Writing doesn’t start after the analysis process, in qualitative research it often should precede, follow and run in parallel to a iterative interpretation.


The standard way to do this is either through a research journal (which is also vital during the data collection process) or through analytic memos. Memos create an important extra level of narrative: an interface between the participant’s data, the researcher’s interpretation and wider theory.


You can also use memos as part of a summary process, to articulate your interpretations of the data in a more concise format, or even throw the data wider and larger by drawing from larger theory.


It’s also a good cognitive exercise: regularly make yourself write what you are thinking, and keep yourself articulating yourself. It will make writing up at the end a lot easier in the end! Memos can be a very flexible tool, and qualitative software can help keep these notes organised. Here are 9 different ways you might use memos as part of your work-flow for qualitative data analysis:

 

Surprises and intrigue
This is probably the most obvious way to use memos: note during your reading and coding things that are especially interesting, challenging or significant in the data. It’s important to do more than just ‘tag’ these sections, reflect to yourself (and others) why these sections or statements stand out.

 

Points where you are not sure
Another common use of memos is to record sections of the data that are ambiguous, could be interpreted in different ways, or just plain don’t fit neatly in to existing codes or interpretations. But again, this should be more than just ‘flagging’ bits that need to be looked at again later, it’s important to record why the section is different: sometimes the act of having to describe the section can help comprehension and illuminate the underlying causation.

 

Discussion with other researchers
Large qualitative research projects will often have multiple people coding and analysing the data. This can help to spread the workload, but also allows for a plurality of interpretations, and peer-checking of assumptions and interpretations. Thus memos are very important in a team project, as they can be used to explain why one researcher interpreted or coded sources in a certain way, and flag up ambiguous or interesting sections for discussion.

 

Paper-trail
Even if you are not working as part of a team, it can be useful to keep memos to explain your coding and analytical choices. This may be important to your supervisors (or viva panel) as part of a research thesis, and can be seen as good practice for sharing findings in which you are transparent about your interpretations. There are also some people with a positivist/quantitative outlook who find qualitative research difficult to trust because of the large amount of seemingly subjective interpretation. Memos which detail your decision making process can help ‘show your working out’ and justify your choices to others.

 

Challenging or confirming theory
This is another common use of memos, to discuss how the data either supports or challenges theory. It is unusual for respondents to neatly say something like “I don’t think my life fits with the classical structure of an Aeschylean tragedy” should this happen to be your theoretical approach! This means you need to make these observations and higher interpretation, and note how particular statements will influence your interpretations and conclusions. If someone says something that turns your theoretical framework on its head, note it, but also use the memos as a space to record context that might be used later to explain this outlier. Memos like this might also help you identify patterns in the data that weren’t immediately obvious.

 

Questioning and critiquing the data/sources
Respondents will not always say what they mean, and sometimes there is an unspoken agenda below the surface. Depending on the analytical approach, an important role of the researcher is often to draw deeper inferences which may be implied or hinted at by the discourse. Sometimes, participants will outright contradict themselves, or suggest answers which seem to be at odds with the rest of what they have shared. It’s also a great place to note the unsaid. You can’t code data that isn’t there, but sometimes it’s really obvious that a respondent is avoiding discussing a particular issue (or person). Memos can note this observation, and discuss why topics might be uncomfrotable or left out in the narrative.


Part of an iterative process
Most qualitative research does not follow a linear structure, it is iterative and researchers go back and re-examine the data at different stages in the process. Memos should be no different, they can be analysed themselves, and should be revisited and reviewed as you go along to show changes in thought, or wider patterns that are emerging.


Record your prejudices and assumptions
There is a lot of discussion in the literature about the importance of reflexivity in qualitative research, and recognising the influence of the non-neutral researcher voice. Too often, this does not go further than a short reflexivity/positionality statement, but should really be a constantly reconsidered part of the analytical process. Memos can be used as a prompt and record of your reflexive process, how the data is challenges your prejudices, or how you might be introducing bias in the interpretation of the data.


Personal thoughts and future directions
As you go through the data, you may be noticing interesting observations which are tangential, but might form the basis of a follow-on research project or reinterpretation of the data. Keeping memos as you go along will allow you to draw from this again and remember what excited you about the data in the first place.

 

 

Qualitative analysis software can help with the memo process, keeping them all in the same place, and allowing you to see all your memos together, or connected to the relevant section of data. However, most of the major software packages (Quirkos included) don’t exactly forefront the memo tools, so it is important to remember they are there and use them consistently through the analytical process.

 

Memos in Quirkos are best done using a separate source which you edit and write your memos in. Keeping your notes like this allows you to code your memos in the same way you would with your other data, and use the source properties to include or exclude your memos in reports and outputs as needed. However, it can be a little awkward to flip between the memo and active source, and there is currently no way to attach memos to a particular coding event. However, this is something we are working on for the next major release, and this should help researchers to keep better notes of their process as they go along. More detail on qualitative memos in Quirkos can be found in this blog post article.

 

 

There is a one-month free trial of Quirkos, and it is so simple to use that you should be able to get going just by watching one of our short intro videos, or the built-in guide. We are also here to help at any stage of your process, with advice about the best way to record your analytical memos, coding frameworks or anything else. Don’t be shy, and get in touch!

 


References and further reading:


Chapman, Y., Francis, K., 2008. Memoing in qualitative research, Journal of Research in Nursing, 13(1). http://jrn.sagepub.com/content/13/1/68.short?rss=1&ssource=mfc

 

Gibbs, G., 2002, Writing as Analysis, http://onlineqda.hud.ac.uk/Intro_QDA/writing_analysis.php

Saldana, J., 2015, The Coding Manual for Qualitative Researchers, Writing Analytic Memos about Narritative and Visual Data, Sage, London. https://books.google.co.uk/books?id=ZhxiCgAAQBAJ

 

 

Starting a qualitative research thesis, and choosing a CAQDAS package

qualitative thesis software

 

For those about to embark on a qualitative Masters or PhD thesis, we salute you!

 

More and more post-graduate students are using qualitative methods in their research projects, or adopting mixed-method data collection and using a small amount of qualitative data which needs to be combined with quantitative data. So this year, how can students decide the best approach for the analysis of their data, and can CAQDAS or QDA software help their studies?

 

First, as far as possible, don’t chose the software, choose the method. Think about what you are trying to research, the best way to get deep data to answer your research questions. The type and amount of data you have will be an important factor. Next, how much existing literature and theory there is around your research area? This will affect whether you will adopt a grounded theory approach, or will be testing or challenging existing theory.

 

Indeed, you may decide that that you don’t need software for your research project. For small projects, especially case studies, you may be more comfortable using printouts of your data, and while reading mark important sections with highlighters and post-it notes. Read Séror (2005) for a comparison of computer vs paper methods. You could also look at the 5 Level QDA, an approach to planning and learning the use of qualitative software so that you develop strategies and tactics that help you make the most of the QDA software.

 

Unfortunately, if you decide you want to use a particular software solution it’s not always as simple as it should be. You will have to eventually make a practical choice based on what software your university has, what support they provide, and what your peers and supervisors use.

 

However, while you are a student, it’s also a good time to experiment and see what works best for you. Not only do all the major qualitative software packages offer a free trial, student licences are hugely discounted against the full versions. This gives you the option to buy a copy for yourself (for a relatively small amount of money).

 

There’s a lot of variety in the different qualitative data analysis software available. The most common one is Nvivo, which your university or department may already have a licence for. This is a very powerful package, but can be intimidating for first-time users. Common alternatives like MAXQDA or Atlas.ti are more user friendly, but also adopt similar spreadsheet-like interfaces. There are also lots of more niche alternatives, for example Transana is unmatched for video analysis, and Dedoose works entirely in the cloud so you can access it from any computer. For a more comprehensive list, check out the Wikipedia list, or the profiles on textanalysis.info

 

Quirkos does a couple of things differently though. First, our student licences don’t expire, and are some of the cheapest around. This means that it doesn’t matter if your PhD takes 3 or 13 years, you will still be able to access your work and data without paying again. And yes, you can keep using your licence into your professional career. It also aims to be the easiest software package to use, and puts visualisations of the data first and foremost in the interface.

 

So give Quirkos a try, but don’t forget about all the other alternatives out there: between them all you will find something that works in the way you want it to and makes your research a little less painful!

 

 

Reflections on qualitative software from KWALON 2016

rotterdam centraal station

Last week saw a wonderful conference held by the the Dutch network for qualitative research KWALON, based at the Erasmus University, Rotterdam. The theme was no less than the future of Qualitative Data Analysis (QDA) software.

 

Chair Jeanine Evers opened the session by outlining 8 important themes the group had identified on qualitative analysis software.

 

The first was the challenge of adding features to software that is requested by users or present in competitors software, without breaking the underlying design of the software. Quirkos really connects to this theme, because we have always tried to have a very simple tool-set, based on a philosophy that the software should be very easy to use. While we obviously take heed of suggestions made by our users, we actually have a comprehensive and limited set of features which we have always planned to introduce, and will continue delivering these over the next few years.

 

However, it is not the intention of Quirkos to become a large software package with lots of features, something Jeanine described as a ‘obese software’ that needs to be put on a diet. It was noted that many software providers have released ‘lite’ versions of their software, and another discussion point was if this fragmented approach can benefit universities and users.

 

User friendliness was another theme of the session, and by keeping Quirkos simple we hope to always have this at the fore of our design philosophy. In my talk (you can now get the slides here) I discussed these themes as mostly being about improving accessibility. To this end, we have tried to make Quirkos not just easier to use, but also to teach and own, with permanent licences and discounts for researchers from  countries that can’t usually afford this type of software. For us, the long-term goal is not just increasing the number of people that use software for qualitative analysis, but the number that are able to take up qualitative research in general.

 

There was also some good discussion at the end of our talk about the risks of making software easy to use: especially that it also makes it easy to use badly. As we’ve discussed many times on this blog, software in general can make it very satisfying to code, and this can appear to be more productive than stepping back and thinking about themes or a undertaking deep readings of the data. These problems can apply to all software packages, so it is important that students and educators work together to learn about the whole analysis procedure, and what part CAQDAS can play.

 

Comments also touched on how memo making is a critical part of a good iterative and reflexive qualitative analysis process: which at the moment Quirkos doesn’t forefront (see for example how F4analyse and a future version of Cassandre will operate). Although it is possible to record memos by typing in a source, which gives you the ability to tag and code your memos, as well as writing notes as source properties, this is currently not highlighted enough and we plan on revamping the memo features in a future update.

 


The final theme of the conference, and a major push, was to promote a standard way to exchange software between qualitative software. At the moment it is very difficult for users to move their coded data from one software package to the other. Although most major packages provide options to export their data to other formats (such as spreadsheet CSV data like Quirkos), there is currently no single standard for how should be formatted, so it is very difficult to bring this data – complete with themes and coding - into another package.

 

There was strong support from the software developers to develop and support such a standard, as well as discussions about existing initiatives such as CATA-XML and QuDEx.


This is very important: but not just for users of different of qualitative analysis software, who want to be able to collaborate with universities and colleagues who use different packages. It’s also important for archival purposes, so that qualitative coded data can be universally shared and stored for secondary analysis, and to make it easier for data to be brought in for analysis from the huge number of digital sources in the digital humanities, such as history, journalism, and social media. Such a standard could also be important for formatting data so that machine learning and natural language processing can automate some of the simpler analysis processes on very large ‘big-data’ datasets.


So there is a lot to be done, but a lot of interest in the area in the next few years, with major and minor players all taking different approaches, and seeking common ground. Quirkos is honoured to be a small part of this, and will do whatever we can to improve the world of qualitative analysis for this and the next generation of researchers.

 

 

Include qualitative analysis software in your qualitative courses this year

teaching qualitative modues

 

A new term is just beginning, so many lecturers, professors and TAs are looking at their teaching schedule for the next year. Some will be creating new courses, or revising existing modules, wondering what to include and what’s new. So why not include qualitative analysis software (also known as CAQDAS or QDA software)?

 

There’s a common misconception that software for qualitative research takes too long to teach, and instructors often aren’t confident themselves in the software (Gibbs 2014), leading to a perception that including it in courses will be too difficult (Rodik and Primorac 2015). It’s also a sad truth that few universities or colleges have support from IT departments or experts when training students on CAQDAS software (Blank 2004).

 

However, we have specifically designed Quirkos to address these challenges, and make teaching qualitative analysis with software simpler. It should be possible to teach the basics of qualitative analysis, as well as provide students with a solid understanding of qualitative software in a one or two hour seminar, workshop or lecture. One of the main aims with Quirkos was to ensure it is easy to teach, as well as learn.

 

With a unique and very visual approach to coding and displaying qualitative data, Quirkos tries to simplify the qualitative analysis process with a reduced set of features and buttons. This means there are fewer steps to go over, a less confusing interface for those starting qualitative analysis for the first time, and fewer places for students to get stuck.

 

To make teaching this as straightforward for educators as possible, we provide free ready-to-use training materials to help educators teach qualitative analysis. We have PowerPoint slides detailing each of the main features and operations. These can be adapted for your class, so you can use some or all of the slides, or even just take the screenshot images and edit the specifics for your own use.

 

Example qualitative data sets are available for use in classes. There are two of these: one very basic set of people talking about breakfast habits and a more detailed one on politics and the Scottish Independence Referendum. With these, you can have complete sources of data and exercises to use in class, or to set a more extensive piece of homework or practical assessed project.

 

We also provide two manuals as PDF files that can be shared as course materials or printed out. There is a full manual, but also a Getting Started guide which includes a step-by-step walkthrough of basic operations, ideal for following in a session. Finally, there are video guides which can be shown as part of classes, or included in links to course materials. These range in length from 5 minute overviews to 1 hour long detailed walkthroughs, depending on the need.

 

There is more information in our blog post on integrating qualitative analysis software into existing curriculums, but it’s also worth remembering that there is a one month free trial for yourself and students. The trial version has all the features with no restrictions, and is identical for students working on Windows, Mac or even Linux.

 

However, if you have any questions about Quirkos and how to teach it, feel free to get in touch. We can tell you about others using Quirkos in their classes, some tips and tricks and any other questions you have on comparing Quirkos to other qualitative analysis software.  You can reach us on Skype (quirkos), email (support@quirkos.com) or by phone during UK office hours (+44 131 555 3736). We’ll always be happy to set up a demo for you: we are all qualitative researchers ourselves, so are happy to share our tips and advice.

 

Good luck for the new semester!

 

Qualitative coding with the head and the heart

qualitative coding head and heart

 

In the analysis of qualitative data, it can be easy to fall in the habit of creating either very descriptive, or very general theoretical codes. It’s often a good idea to take a step back, and examine your coding framework, challenging yourself to look at the data in a fresh way. There are some more suggestions for how to do this in a blog post article about turning coding strategies on their head. But while in Delhi recently to deliver some training on using Quirkos, I was struck by a couple of exhibits at the National Museum which in a roundabout way made me think about coding qualitative data, and getting the balance right between analytical and emotional coding frameworks.

 

There were several depictions of Hindu deities trampling a dwarf called Apasmāra, who represented ignorance. I loved this focus of minimising ignorance, but it’s important to note that in Hindu mythology, ignorance should not be killed or completely vanquished, lest knowledge become too easy to obtain without effort.

 

Another sculpture depicted Yogini Vrishanna, a female deity that had taken the bull-head form. It was apparently common for deities to periodically take on an animal head to prevent over-intellectualism, and allow more instinctive, animalistic behaviour!

 

I was fascinated between this balance being depicted between venerating study and thought, but at the same time warning against over thinking. I think this is a message that we should really take to heart when coding qualitative data. It’s very easy to create coding themes that are often far too simple and descriptive to give much insight in the data: to treat the analysis as purely a categorization exercise. When this happens, students often create codes that are basically an index of low-level themes in a text. While this is often a useful first step, it’s important to go beyond this, and create codes (or better yet, a whole phase of coding) which are more interpretive, and require a little more thinking.

 

However, it’s also possible to go too far in the opposite direction and over-think your codes. Either this comes from looking at the data too tightly, focusing on very narrow and niche themes, or from the over-intellectualising that Yogini Vrishanna was trying to avoid above. When the researcher has their head deeply in the theory (and lets be honest this is an occupational hazard for those in the humanities and social sciences), there is a tendency to create very complicated high-level themes. Are respondents really talking about ‘social capital’, ‘non-capitalocentric ethics’ or ‘epistemic activism’? Or are these labels which the researcher has imposed on the data?

 

These might be the times we have to put on our imaginary animal head, and try to be more inductive and spontaneous with our coding. But it also requires coding less from the head, and more from the heart. In most qualitative research we are attempting to understand the world through the perspective of our respondents, and most people are emotional beings, acting not just for rational reasons.

 

If our interpretations are too close to the academic, and not the lived experiences of our study communities, we risk missing the true picture. Sometimes we need a this theoretical overview to see more complex trends, but they should never be too far from the data in a single qualitative study. Be true to both your head and your heart in equal measure, and don’t be afraid to go back and look at your data again with a different animal head on!

 

If you need help to organise and visualise all the different stages of coding, try using qualitative analysis software like Quirkos! Download a free trial, and see for yourself...

 

10 tips for sharing and communicating qualitative research

sharing qualitative research data


Writing up and publishing research based on qualitative or mixed methods data is one thing, but most researchers will want to go beyond this, and engage with the wider public and decision makers. This requires a different style of publication, and a different style of writing. We are not talking about journal articles, funders reports, book chapters or a thesis here, but creating short, engaging and impactful summaries of your research that anyone can read and share. Research that just sits on a shelf doesn't change the world!

 

The aim here is primarily outreach and impact – making sure that your research is read and has applications beyond academia, especially to the general public and decision makers. These could be local or national government, NGOs, funding bodies or service providers. It may even be that your research has implications for a particular group of people, for example those with a particular health condition, demographic group, or work in a certain field. There’s also the increasingly common expectation to should share results with participants.

 

Generally speaking, outputs and reports for these groups of people should be short and use non-technical language: it’s not enough to just provide with a copy of your thesis or a journal article. Those are almost always written for a very specific audience, academics, and are difficult to read for the general public. Outside academia, most don’t have access to journals which often require subscriptions, and some government departments have cut down on library services in these areas.

 

In the research I’ve been involved with, we’ve even created short summaries of our research for GPs, clinicians, and health managers: people who are certainly familiar with journal articles, but rarely have the time to read them. It became clear in our discussions with people we were trying to engage with that one page summaries were the best format to get findings read.

 

It sounds like a lot of extra work at the end of a research project on top of publications, but this can be just as important, possibly even an ethical requirement. Some funding boards and IRBs require the creation of research outputs for lay audiences.

 

There are plenty of guides to help with writing up qualitative data for articles and book chapters (eg Ryan 2006), but what about writing up qualitative research for non-academic audiences? I’ve written about the challenges of explaining qualitative data before, but these tips below contain more specific advice on creating short, engaging summaries for the general public. They also provide prompts to make you think about promoting your research so it is read by more people, and has more impact.

 


1. Create specific outputs for each audience

It maybe that there are different groups of people you want to reach: the general public, politicians, or experts in a particular field. Consider creating several short outputs targeted at each one. A short summary written for a lay audience might not have the level of detail a government body would want to see, and you might also want to highlight findings which are interesting to certain readers.

 

Think about the different groups of people you want to engage with, and draw up a list of what outputs would work best for each one. These don’t have to be written either, it could be a coffee morning, presenting at a meeting, or a short video. Choose a format that works best for each type of audience, and the most important message to get across to each.

 


2. Link to topical issues

Qualitative research often takes a very in-depth approach to a specific research question, but this depth also means that it can engage with wider but connected themes. It maybe that your research is already on something topical, such as diet or social media. But there maybe important local issues your research can feed into, or wider problems that your research project illustrates a small part of. Engaging with a currently trending issue can not only get more views, but hopefully improve the quality of debate.

 

Where possible, try and consider issues that are not just part of a short-term media cycle, but are longer trends likely to come up again and again, such as house prices or obesity. It’s not necessary to twist your main finding to make them fit, just find a relevant connection.

 


3. Tell the story

I think this is the key to communicating qualitative data. People engage with stories about people better than they do with cold reports and statistics. That is why media tends to focus on individual politicians and celebrities more than their policies or what they do, and is also a better way to retain information. Give your stories context and causality (because of this, that happened to this person), and you are following the same basic rules for good storytelling that scriptwriters and novelists follow.

 

While you can take this in very creative directions, for example creating an animated video story based on one of your participants, it maybe that a box containing a case study is enough to provide a report with illuminating context. Stories are the most powerful part of qualitative research, make sure you use them!

 


4. Make them visual, and beautiful

People are more likely to pick up and engage with visual outputs, and pictures or other visual elements help people understand and associate with the findings. Try not to rely on generic clip-art or stock photos, choose images that are unique and specific to your work. If it’s not appropriate to include pictures of respondents or the local area, consider talking to artists or photographers that could create or let you use something more abstract.

 

It’s also important to consider how a research output looks: a written report shouldn’t just be a wall of long text, make sure it is broken up, and has plenty of white-space. It may even be worth getting a designer involved: this often doesn’t cost too much, but can make the output look a lot more professional, tempting to pick up and easier to follow. The same goes for presentations and events too!

 

Think about creating visualisations of your qualitative data, rather than just a series of quotes. Qualitative analysis software can help with this by making things like word-clouds or visualisations of important findings and connections in the research. Quirkos tries to make visualisations like this easy to understand and export, so check out the rest of the website to see how it can help with qualitative analysis.

 


5. Explain the methods (but briefly)

When presenting qualitative data, you should consider the fact that many people aren’t familiar with qualitative research or the methods you might have used. Those more used to quantitative data (especially in the public sector) might consider that your sample size is too small, or your research findings aren’t rigorous enough. It can be worth pre-empting these criticisms, but not by being apologetic. Don’t just say that ‘this is a limited study’ or ‘further research is needed’, be positive about the depth of your investigation, how commonly used your methods are, and if appropriate show how your work contradicts with or supports other research.

 

Have a short section describing your research methods, but don’t provide too much detail. If the reader is interested in this, provide a way for them to read more somewhere else, for example a publication or a project website. It’s better to tease the reader and make them want more, rather than providing too much detail in the first place. Speaking of which:

 


6. Stay away from the academic debates

Generally, this is not the place for debates on epistemology, ontology or what other academics are saying in the field. While it is sometimes possible to explain these issues with non-technical language, it is probably not something that this audience is interested in.

 

This can be hard if your research question was specifically focused around, say, a Foucauldian interpretation of language used to reviews of artisan coffee shops, but focus on the findings that are of interest to the general public. This is why just creating a summary of a journal article or full report generally doesn’t work well for a general output.

 


7. Write for others

Writing a popular output is one thing, but finding  readers is hard. So why not find channels that already engage with the group of people you want to reach, and write for them instead. This could be a popular blog on a health condition, a trade magazine or even something for the popular press. Approach these people and ask if they would be interested in a piece about your research, stressing why it would be interesting to their readers. Some are glad for the opportunity to have something to fill space, and if you can demonstrate relevance to a topical issue, journalists might get involved as well.

 

If you are considering going down the media route, your university may have a press office that could help create press releases, and suggest the right editors and provide training to researchers on being interviewed on radio or TV. Just make sure that the outreach is serving your agenda, not just promoting the university or trying to spark controversy.

 

Look for relevant events you could present at such as workshops and conferences, since these can be targeted at professionals like health workers, not just academics.

 

 

8. Promote your outputs

It’s not good enough to create a report, stick it on your own website and forget about it. You need to promote your outputs and make sure that people can find them. Promotion should also be audience specific: where are my readers, and what do they already engage with? If you are running an event, should you have posters in cafés, or an advert in the local paper?

 

If you have a project website, this needs to be promoted too. Make sure people can find it if they are searching for issues around your research, and ask other websites in the area to put in a link to your web page. Keywords are important too: what searches are your readers going to make? It’s probably not “qualitative research on peer support for cancer” but “support groups for cancer”. Make sure the right terms appear on your website and as the heading for your outputs.

 

9. Make them long-term accessible

A one off event or report is great, but only can target people currently looking to engage with your research. Policy makers change year after year, and with health issues, new people will be diagnosed all the time and will look for information.

 

If you have a project website, you need to consider a long-term strategy for it: make sure it is accessible for a long period of time, and can be updated. The right people should have physical copies of reports, that way they can access them later. There also might be good places to keep distributing summaries, like libraries, community centres or GP surgeries.

 

It’s also worth coming back to your project and promoting it after a period of time. This is difficult in academia where funding is research and time limited, but set aside a one and two year anniversary and spend time reaching out to new people. Impact and engagement is important in academia, and a fresh attempt at reaching out after a period of time can dramatically increase the number of people reached.

 

 

10. Don’t forget social media!

Social media can be a good way to promote your research, as it is fairly easy to find people from the right audience. They may be following a particular person in the field, or declare an interest in a relevant hobby or workplace. Try Linkedin to contact people working in a certain field, or Facebook for getting out to the general public.

 

However, it is also possible to share findings in social media too. A Tweet is a very small amount of text for a qualitative research project, but is enough to tease a finding and provide a link containing more information. You can also create outputs in the form of infographics, pictures and video which people can share with others.

 

Creative and varied outputs are more likely to get general engagement, so experiment: make your materials fun and stand out from the crowd to get the word out!

 

 

Making qualitative analysis software accessible

accessible qualitative analysis software


Studies and surveys seem to show that the amount of qualitative research is growing, and that more and more people are using software to help with their qualitative analysis (Woods et al. 2015). However, these studies also highlight that users report problems with learning qualitative software, and that universities sometimes struggle to provide enough expertise to teach and troubleshoot them (Gibbs 2014).


Quirkos was specifically designed to address many of these issues, and our main aim is to be a champion for the field of qualitative research by making analysis software more accessible. But what does accessibility mean in this context, and what problems still need to be overcome?

 

Limitations of paper

The old paper and highlighters method is a very easy and accessible approach to qualitative analysis. Indeed, it’s common for some stages in the analysis exploration to be done on paper (such as reviewing), even if most of it is done in software. However, when projects get above a certain size or complexity, it can be difficult to keep track of all the different sources and themes. Should you have dozens of topics you are looking for in the project, you can quickly run out of different colours for your highlighters or Post-it notes (6 colours seems to be the most you can easily find) and I’ve seen very creative but laborious use of alternating coloured stripes and other techniques!

 

In these situations, qualitative analysis software can actually be more accessible, and make the process a lot easier. The big advantage to computers is that they have huge memories, and think nothing of working with hundreds of sources, and hundreds of coding topics. There are some people that are able to keep hundreds of topics in their head at once, (my former boss Dr Sarah Salway was one of these) but for us mortals, software can really help. However, software needs to try and be as easy to use as paper, and make sure that it doesn’t start making the data more difficult to see, or makes the coding process seem more important than deep reading and comprehension.

 

Learning curve

Secondly, if the software is going to be accessible, it has to be easy to learn and understand. While the best way to learn is often with face-to-face teaching, not everyone has the luxury of access to this, and it can be expensive. So there needs to be good, and freely available training materials. Ideally the software would be so simple that it didn’t need tuition at all, but inevitably people will get stuck, and good video guides and manuals should be easily available.


The software has to tread a fine line between being clear and non-patronising. I did have a discussion with one trainer in qualitative analysis about introducing an animated guide like Clippy to QDA software, to guide users through the process. Can you imagine what this would be like? A little character that pops up and says things like “Hi! It looks like you are doing grounded theory! Would you like some help with that?”. But most users I talk to want the software to be as invisible as possible. If it gets in the way frequently it is hindering, not helping the analysis process.

 

Flexibility

Software also needs to be as flexible as possible, it’s no good if it doesn’t fit your approach or the way you need to work. So it has to allow you to work with the type of data you have, without having to spend ages reformatting it.
It should be neutral to your approach as well, making sure that whatever the methodological and theoretical approach the user is taking, the software will allow researchers to work their own way. A lot of flexibility requirements comes when working with others too, getting data both in and out should be painless, and fit the rest of a researcher’s workflow.

 

Sharing with others

Most qualitative researchers like to work with others, either as part of a research team, or just as a resource to bounce ideas off. Sending project files from qualitative analysis software to another research is easy enough, but often only if they are using the same software on the same operating system. Cross platform working is really important, and it is frustrating at the moment how difficult it is to get coded data from one software package to another. We are having discussions with other developers of qualitative software about making sure that there is interoperability, but it is going to be a long journey.


It’s also important for software to create exports of the data in more common formats, such as PDF, Word files or the like, so people without specialist CAQDAS software can still engage and see the data.

 

Visual impairment

At the moment Quirkos is a very visual piece of software, and not well suited to those with visual or physical limitations. We have tried to choose options that make the software easier for those with vision impairment, such as high contrast text and large font sizes, but there is still a long way to go. At the moment, although shortcut keys can make using Quirkos a lot easier, navigating and selecting text without a mouse is not possible. We want to add the ability to run all the main operations from the keyboard or a specialist controller so that there are fewer barriers for those with reduced mobility.


We’ve even had serious discussions with blind qualitative researchers about how Quirkos could meet their needs! The main problem here seems to be the wide range of specialist computers and equipment – although there are fantastic tools out there for people with total or near-total visual impairment, they are all very different, and getting software that would work with them all is a huge challenge.

 

Affordability

However, there is another barrier to access for many: the price of software licences. In many countries, relative low wages mean that qualitative analysis software is prohibitively expensive. This is not just in Latin America, Africa and many parts of Asia: even in Eastern Europe, a single CAQDAS licence can cost as much as many earn in a month. (Haratyk and Kordasiewicz 2014).


So also am proud to announce from today, that we will offer a 25% discount for researchers based in ‘developing’ or ‘emerging’ nations. I don’t like these terms, so for clarity I am taking this to mean any country with a GDP Per-Capita below US$2600 PPP, or a monthly average salary below 1000EUR. This is on top of our existing discounts for students, education, charity and education sectors. As far as I can see, we are the first qualitative analysis software company to offer a discount of this type. To check if your country qualifies, and to place an order with this discount, just send an e-mail to sales@quirkos.com and we will be happy to help.


Quirkos is already around half the price of the other major CAQDAS software packages, but from now we are able to provide an extra discount to researchers in 150 countries, representing nearly 80% of the world population. We hope this will help qualitative researchers in these countries to use qualitative research to explore and answer difficult questions in health, development, transparency and increasing global happiness.

 

Reaching saturation point in qualitative research

saturation in qualitative research

 

A common question from newcomers to qualitative research is, what’s the right sample size? How many people do I need to have in my project to get a good answer for my research questions? For research based on quantitative data, there is usually a definitive answer: you can decide ahead of time what sample size is needed to gain a significant result for a particular test or method.

 

This post is hosted by Quirkos, simple and affordable qualitative analysis software. Download a one-month free trial today!

 

In qualitative research, there is no neat measure of significance, so getting a good sample size is more difficult. The literature often talks about reaching ‘saturation point’ - a term taken from physical science to represent a moment during the analysis of the data where the same themes are recurring, and no new insights are given by additional sources of data. Saturation is for example when no more water can be absorbed by a sponge, but it’s not always the case in research that too much is a bad thing. Saturation in qualitative research is a difficult concept to define Bowen (2008), but has come to be associated with the point in a qualitative research project when there is enough data to ensure the research questions can be answered.

 

However, as with all aspects of qualitative research, the depth of the data is often more important than the numbers (Burmeister & Aitken, 2012). A small number of rich interviews or sources, especially as part of a ethnography can have the importance of dozens of shorter interviews. For Fusch (2015):

 

“The easiest way to differentiate between rich and thick data is to think of rich as quality and thick as quantity. Thick data is a lot of data; rich data is many - layered, intricate, detailed, nuanced, and more. One can have a lot of thick data that is not rich; conversely, one can have rich data but not a lot of it. The trick, if you will, is to have both.”

 

So the quantity of the data is only one part of the story. The researcher needs to engage with it at an early level to ensure “all data [has] equal consideration in the analytic coding procedures. Frequency of occurrence of any specific incident should be ignored. Saturation involves eliciting all forms of types of occurrences, valuing variation over quantity.” Morse (1995). When the amount of variation in the data is levelling off, and new perspectives and explanations are no longer coming from the data, you may be approaching saturation. The other consideration is when there are no new perspectives on the research question, for example Brod et al. (2009) recommend constructing a ‘saturation grid’ listing the major topics or research questions against interviews or other sources, and ensuring all bases have been covered.

 

But despite this, is it still possible to put rough numbers on how many sources are required for a qualitative research project? Many papers have attempted to do this, and as could be expected, the results vary greatly. Mason (2010) looked at the average number of respondents in PhD thesis using on qualitative research. They found an average of 30 sources were used, but with a low of 1 source, a high of 95 and a standard deviation of 18.5! It is interesting to look at their data tables, as they show succinctly the differences in sample size expected for different methodological approaches, such as case study, ethnography, narrative enquiry, or semi-structured interviews.

 


While 30 in-depth interviews may seem high (especially for what is practical in a PhD study) others work with much less: a retrospective examination from a qualitative project by Guest et al. (2006) found that even though they conducted 60 interviews, they had saturation after 12, with most of the themes emergent after just 6. On the other hand, if students have supervisors who have more of a mixed-method or quantitative background, they will often struggle to justify the low number of participants suggested for methods of qualitative enquiry.

 


The important thing to note is that it is nearly impossible for a researcher to know when they have reached saturation point unless they are analysing the data as it is collected. This exposes one of the key ties of the saturation concept to grounded theory, and it requires an iterative approach to data collection and analysis. Instead of setting a fixed number of interviews or focus-groups to conduct at the start of the project, the investigator should be continuously going through cycles of collection and analysis until nothing new is being revealed.

 


This can be a difficult notion to work with, especially when ethics committees or institutional review boards, limited time or funds place a practical upper limit on the quantity of data collection. Indeed Morse et al (2014) found that in most dissertations they examined, the sample size was chosen for often practical reasons, not because a claim of saturation was made.

 


You should also be aware that many take umbrage at the idea that one should use the concept of saturation. O’Reilly (2003) notes that since the concept of saturation comes out of grounded theory, it’s not always appropriate to apply to research projects, and the term has become over used in the literature. It’s also not a good indicator by itself of the quality of qualitative research.

 


For more on these issues, I would recommend any of the articles referenced above, as well as discussion with supervisors, peers and colleagues. There is also more on sampling considerations in qualitative research in our previous blog post article.

 

 

Finally, don’t forget that Quirkos can help you take an iterative approach to analysis and data collection, allowing you to quickly analyse your qualitative data as you go through your project, helping you visualise your path to saturation (if you so choose this approach!). Download a free trial for yourself, and take a closer look at the rest of the features the software offers.