Making the most of bad qualitative data

 

A cardinal rule of most research projects is things don’t always go to plan. Qualitative data collection is no difference, and the variability in approaches and respondents means that there is always the potential for things to go awry. However, the typical small sample sizes can make even one or two frustrating responses difficult to stomach, since they can represent such a high proportion of the whole data set.


Sometimes interviews just don’t go well: the respondent might only give very short answers, or go off on long tangents which aren’t useful to the project. Usually the interviewer can try and facilitate these situations to get better answers, but sometimes people can just be difficult. You can see this in the transcript of the interview with ‘Julie’ in the example referendum project. Despite initially seeming very keen on the topic, perhaps she was tired on the day, but cannot be coaxed into giving more than one or two word answers!


It’s disappointing when something like this happens, but it is not the end of the world. If one interview is not as verbose or complete as some of the others it can look strange, but there is probably still useful information there. And the opinions of this person are just as valid, and should be treated with the same weight. Even if there is no explanation, disagreeing with a question by just saying ‘No’ is still an insight.


You can also have people who come late to data collection sessions, or have to leave early resulting in incomplete data. Ideally you would try and do follow up questions with the respondent, but sometimes this is just not possible. It is up to you to decide whether it is worth including partial responses, and if there is enough data to make inclusion and comparison worthwhile.


Also, you may sometimes come across respondents who seem to be outright lying – their statements contradict, they give ridiculous or obviously false answers, or flat out refuse to answer questions. Usually I would recommend that these data sources are included, as long as there is a note of this in the source properties and a good justification for why the researcher believes the responses may not be trusted. There is usually a good reason that a respondent chooses to behave in such a way, and this can be important context for the study.


In focus group settings there can sometimes be one or two participants who derail the discussion, perhaps by being hostile to other members of the group or only wanting to talk about their pet topics and not the questions on the table. This is another situation where practice at mediating and facilitating data collection can help, but sometimes you just have to try and extract whatever is valuable. But organising focus groups can be very time consuming, and consume so many potentially good respondents in one go, so having poor data quality from one of the sessions can be upsetting. Don’t be afraid to go back to some of the respondents and see if they would do another smaller session, or one-on-ones to get more of their input.


However, the most frustrating situation is when you get disappointing data from a really key informant: someone that is an important figure in the field, is well connected or has just the right experience. These interviews don’t always go to plan, especially with senior people who may not be willing to share, or have their own agenda in how they shape the discussion. In these situations it is usually difficult to find another respondent who will have the same insight or viewpoint, so the data is tricky to replace. It’s best to leave these key interviews until you have done a few others; that way you can be confident in your research questions, and will have some experience in mediating the discussions.


Finally, there is also lost data. Dictaphones that don’t record or get lost. Files gone missing and lost passwords. Crashed computers that take all the data with them to an early and untimely grave! These things happen more often than they should, and careful planning, precautions and backups are the only way to protect against these.


But often the answer to all these problems is to collect more data! Most people using qualitative methodologies should have a certain amount of flexibility in their recruitment strategy, and should always be doing some review and analysis on each source as it is collected. This way you can quickly identify gaps or problems in the data, and make sure forthcoming data collection procedures cover everything.


So don’t leave your analysis too late, get your data into an intuitive tool like Quirkos, and see how it can bring your good and bad research data to light! We have a one month free trial, and lots of support and resources to help you make the most of the qualitative data you have. And don’t forget to share your stories of when things went wrong on Twitter using the hashtag #qualdisasters!

 

Practice projects and learning qualitative data analysis software

image by zaui/Scott Catron

 

Coding and analysing qualitative data is not only a time consuming, it’s a difficult interpretive exercise which, like learning a musical instrument, gets much better with practice. However, lots of students starting their first major qualitative or mixed method research project will benefit from completing a smaller project first, rather than starting by trying to learn a giant symphony. This will allow them to get used to qualitative analysis software, working with qualitative data, developing a coding framework and getting a realistic expectation of what can be done in a fixed time frame. Often people will try and learn all these aspects for the first time when they start a major project like a masters or PhD dissertation, and then struggle to get going and take the most effective approach.

 

Many scholars, including those advocating the 5 Level QDA approach suggest that learning the capabilities of the software and qualitative data separately, since one can effect the other. And a great way to do this is to actually dig in and get started with a separate smaller project. Reading textbooks and the literature can only prepare you so much (see for example this article on coding your first qualitative data), but a practical project to experiment and make mistakes in is a great preparation for the main event.

 

But what should a practice project look like? Where can I find some example qualitative data to play with? A good guideline is to take just a few sources, even just 3 or 4 from a method that is similar to the data collection you will use for your project. For example, if you are going to have focus groups, try and find some already existing focus group transcripts to transcribe. Although this can be daunting, there are actually lots of ways to quickly find qualitative data that will not only make you more familiar with real qualitative data, but also the analysis process and accompanying software tools. This article gives a couple of suggestions for a mini project to hone your skills!

 


News and media

A quick way to practice your basic coding skills is to do a small project using news articles. Just choose a recent (or historical) event, collect a few articles either from different news websites or over a period of time. Looking at conflicts in how events are described can be revealing, and is good for getting the right analytical eye to examine differences from respondents in your main project. It’s easy to go to different major news websites (like the Telegraph, Daily Mail, BBC News or the NYT) and copy and paste articles into Quirkos or other qualitative analysis software. All these sites have searchable archives, so you can look for a particular topic and find older articles.

 

Set yourself a good research question (or two), and use this project to practice generating a coding framework and exploring connections and differences across sources.

 

 

Qualitative Data Archives

If you want some more involved experience, browse some of the online repositories of qualitative data. These allow you to download the complete data set from research projects large and small. Since much government (or funding board) funded research requires data to be made publicly available, there are increasing numbers of data sets available to download which make a great way to look at real qualitative data, and practice your analysing skills. I’ll share two examples here, the first is the UK Data Archive and the second the Syracuse Qualitative Data Repository.

 

Regardless of where you are based, these resources offer an amazing variety of data types and topics. This can make your practice fun – there are data sets on some fascinating and obscure areas, so choose something interesting to you, or even something completely new and different as a challenge. You also don’t have to use all the sources from a large project – just choose three or four to start with, you can always add more later if you need extra coding experience.

 

 

Literature reviews

Actually, qualitative analysis software is a great way to get to grips with articles, books and policy documents relating to your research project. Since most people will want to do a systematic or literature review before they start a project, bringing your literature into qualitative software is a good way to learn the software while also working towards your project. While reading through your literature, you can create codes/themes to describe key points in theory or previous studies, and compare findings from different research projects.

In Quirkos it is easy to bring in PDF articles from journals or ebooks, and then you will have not only a good reference management system, but somewhere you can keep the full text of relevant articles, tagged and coded so you can find key quotes quickly for writing up. Our article here gives some good advice on using qualitative software for systematic and literature reviews.

 

 


Our own projects

Quirkos also has made two example projects freely available for learning qualitative analysis with any software package. The first is great for beginners, a fictional project about healthy eating options for breakfast. These 6 sources are short, but with rich information, so can be fully coded in less than an hour. Secondly, we conduded a real research project on the Scottish Referendum for Independence, and 12 transcribed semi-structured interviews are made available for your own practice and interpretation.

 

The advantage of these projects is that they both have fully coded project files to use as examples and comparison. It’s rare to find people sharing their coding (especially as an accessible project file) but can be a useful guide or point of comparison to your own framework and coding decisions.

 

download Quirkos qualitative research software

 

Ask around

Talk to faculty in your department and see if they have any example data sets you can use. Some academics will already have these for teaching purposes or taken from other research projects they are able to share.

 

It can also be a good exercise to do a coding project with someone else. Regardless of which option you choose from the example qualitative data sources above, share the data with another student or colleague, and go and do your own coding  separately. Once you are both done, meet up and compare your results – it will be really revealing to see how different people interpreted the data, how their coding framework looked, and how they found working with the software. It’s also good motivation and time management to have to work to a mutually set deadline!

 

 

The great thing about starting a small learning project is that it can be a perfect opportunity to experiment with different qualitative analysis software. It may be that you only have access to one option like Nvivo, MAXQDA, or Atlas.Ti at your institution, but student licences are very cheap and affordable, so make a great option for learning qualitative analysis. All the major packages have a free trial, so you can try several (or them all!) and find out which one works best for you. Doing this with a small example project lets you practice key techniques and qualitative methods, and also think through how best to collect and structure your data for analysis.

 

Quirkos probably has the best deal for qualitative research software, for example our student licences are cheap at just $59 (£49 or €58) and don’t expire. Most of the other packages only give you six months or a year but we let you use Quirkos as long as you need, so you will always be able to access your data – even after you graduate. Even academics and supervisors will find that Quirkos is much more affordable and easier to learn. Of course, there is a no obligation or registration trial, and all our support and training materials are free as well. So make sure you make the most informed decision before you start your research, and we hope that Quirkos becomes your instrument of choice for qualitative analysis!

 

 

Looking back and looking forward to qualitative analysis in 2017

2017 in qualitative analysis software - Janus

 

In the month named for Janus, it’s a good time to look back at the last year for Quirkos and qualitative analysis software and look forward to new developments for 2017.

 

It’s been a good year of growth for Quirkos, we now can boast of users in more than 100 universities across the world. But we can see how many more people are using Quirkos in these institutions as the word grows. There is no greater complement than when researchers recommend Quirkos to their peers, and this has been my favourite thing to see this year.

 

We also we honoured to take part in many international conferences and events, including TQR in January, ICQI in May, KWALON in August and QDR in October. Next year already has many international events on the calendar, and we hope to be in your neck of the woods soon! We have also run training workshops in many universities across the UK, and demand ensures these will continue in 2017.

 

Our decision to offer a 25% discount to researchers in developing countries has opened the door to a lot of interest and we are helping many researchers use qualitative analysis software for the first time.

 

The blog has also become a major resource for qualitative researchers, with more than 110 posts and counting now archived on the site, attracting thousands of visitors a month. In the next year we will be adding some new experimental formats and training resources to complement our methodology articles.

 

In terms of the Quirkos software itself, 2016 saw our most major upgrade to date, v1.4 which brought huge improvements in speed for larger projects. In early 2017 we will release a minor update (v1.4.1) which will provide a few bug fixes. We are already working towards v1.5 which will be released later in the year and add some major new requested features and refinements, but keep the same simple interface and workflow that people love.

 

We also have a couple of major announcements in the next month about the future of qualitative analysis software, Quirkos and the new skills we will be bringing on board. Watch this space!