We live in a world of deep qualitative data.

It’s often proposed that we are very quantitatively literate. We are exposed to numbers and statistics frequently in news reports, at work, when driving, with fitness apps etc. So we are actually pretty good at understanding things like percentages, fractions, and making sense of them quickly. It’s a good reason why people like to see graphs and numerical summaries of data in reports and presentations: it’s a near universal language that people can quickly understand.

But I believe we are also really good at qualitative understanding.

Bohn and Short in a 2009 study estimated that “The average American consumes 100,500 words of information in a single day”, comprised of conversations, TV shows, news, written articles, books… It sounds like a staggering amount of qualitative data to be exposed to, basically a whole PhD thesis every single day!

Obviously, we don’t digest and process all of this, people are extremely good at filtering this data; ignoring adverts, skim reading websites to get to the articles we are interested in and skim reading those, and of course, summarising the gist of conversations with a few words and feelings. That’s why I argue that we are nearly all qualitative experts, summarising and making connections with qualitative life all the time.

And those connections are the most important thing, and the skill that socially astute humans do so well. We can pick up on unspoken qualitative nuances when someone tells us something, and understand the context of a news article based on the author and what is being reported. Words we hear such as ‘economy’ and ‘cancer’ and ‘earthquake’ are imbued with meaning for us, connecting to other things such as ‘my job’ and ‘fear’ and ‘buildings’.

This neural network of meaning is a key part of our qualitative understanding of the world, and whether we want to challenge these by some type of Derridan deconstruction of our associations between language and meaning, they form a key part of our daily prejudices and understanding of the world in which we live.

For me, a key problem with qualitative analysis is that it struggles to preserve or record these connections and lived associations. I touched on this issue of reductionism in the last blog post article on structuring unstructured qualitative data, but it can be considered a major weakness of qualitative analysis software. Essentially, one removes these connected meanings from the data, and puts it as a binary category, or at best, represents it on a scale.

Incidentally, this debate about scaling and quantifying qualitative data has been going on for at least 70 years from Guttman, who even in this 1944 article notes that there has been ‘considerable discussion concerning the utility of such orderings’. What frustrates me at the moment is that while some qualitative analysis software can help with scaling this data, or even presenting it in a 2 or 3 dimensional scale by applying attributes such as weighting, it still is a crude approximation of the complex neural connections of meaning that deep qualitative data possesses.

In my experiments getting people with no formal qualitative or research experience to try qualitative analysis with Quirkos, I am always impressed at how quickly people take to it, and can start to code and assign meaning to qualitative text from articles or interviews. It’s something we do all the time, and most people don’t seem to have a problem categorising qualitative themes. However, many people soon find the activity restrictive (just like trained researchers do) and worry about how well a basic category can represent some of the more complex meanings in the data.

Perhaps one day there will be practical computers and software that ape the neural networks that make us all such good qualitative beings, and can automatically understand qualitative connections. But until then, the best way of analysing data seems to be to tap into any one of these freely available neural networks (i.e. a person) and use their lived experience in a qualitative world in partnership with a simple software tool to summarise complex data for others to digest.

After all, whatever reports and articles we create will have to compete with the other 100,000 words our readers are consuming that day!

The terms ‘unstructured data’ and ‘qualitative data’ are often used interchangeably, but unstructured data is becoming more commonly associated with data mining and big data approaches to text analytics. Here the comparison is drawn with databases of data where we have a defined field and known value and the loosely structured (especially to a computer) world of language, discussion and comment. A qualitative researcher lives in a realm of unstructured data, the person they might be interviewing doesn’t have a happy/sad sign above their head, the researcher (or friend) must listen and interpret their interactions and speech to make a categorisation based on the available evidence.

At their core, all qualitative analysis software systems are based around defining and coding: selecting a piece of text, and assigning it to a category (or categories). However, it is easy to see this process as being ‘reductionist’: essentially removing a piece of data from it’s context, and defining it as a one-dimensional attribute. This text is about freedom. This text is about liberty. Regardless of the analytical insight of the researcher in deciding what relevant themes should be, and then filtering a sentence into that category, the final product appears to be a series of lists of sections of text.

This process leads to difficult questions such as, is this approach still qualitative? Without the nuanced connections between complicated topics and lived experiences, can we still call something that has been reduced to a binary yes/no association qualitative? Does this remove or abstract researchers from the data? Isn't this a way of quantifying qualitative data?

While such debates are similarly multifaceted, I would usually argue that this process of structuring qualitative data does begin to categorise and quantify it, and it does remove researchers from their data. But I also think that for most analytical tasks, this is OK, if not essential! Lee and Fielding (1996) say that “coding, like linking in hypertext, is a form of data reduction, and for many qualitative researchers is an important strategy which they would use irrespective of the availability of software”. When a researcher turns a life into 1 year ethnography, or a 1 hour interview, that is a form of data reduction. So is turning an audio transcript into text, and so is skim reading and highlighted printed versions of that text.

It’s important to keep an eye on the end game for most researchers: producing a well evidenced, accurate summary of a complex issue. Most research, as a formula to predict the world or a journal article describing it, is a communication exercise that (purely by the laws of entropy if not practicality) must be briefer than the sum of it’s parts. Yet we should also be much more aware that we are doing this, and together with our personal reflexivity think about the methodological reflexivity, and acknowledge what is being lost or given prominence in our chosen process.

Our brains are extremely good at comprehending the complex web of qualitative connections that make everyday life, and even for experienced researchers our intuitive insight into these processes often seem to go beyond any attempt to rationalise them. A structuralist approach to qualitative data can not only help as an aide-mémoir, but also to demonstrate our process to others, and challenge our own assumptions.

In general I would agree with Kelle (1997) that “the danger of methodological biases and distortion arising from the use of certain software packages is overemphasized in current discussions”. It’s not the tool, it’s how you use it!

On the 23rd and 24th of June, we are running a series of workshops in Sheffield: both at the University of Sheffield, and Sheffield Hallam University. The events are open to students, academics and anyone from the public sector or commercial areas who has an interest in analysing qualitative text.

The first session is on the Tuesday 23rd at 10am, in the Hicks building at the University of Sheffield. In the afternoon there is a session from 2pm at the city-centre campus of Sheffield Hallam University, in the Cantor building. Finally, for those in Sport, and Health and Social Care, there is a workshop on the morning of the 24th at 9:30 in the Mews building in Collegiate Campus.

Hopefully this gives a convenient location for everyone! The interest has been really good already, and although the workshops are all free, and cover the same material, you do need to book in advance to help us get a hold of numbers. Just e-mail info@quirkos.com and we will reserve you a space. A lot more information on the workshops are available on our workshops page.

These workshops have run in many other cities and institutions now, and in under 2 hours, you can go from never using Quirkos before to installing and running it on your laptop, bringing in data, coding and producing reports: in short all the steps you need for qualitative text analysis. It's helpful if you have some idea what is involved in qualitative analysis, but not essential. We can also provide laptops on the day if you need to borrow one.

These courses should be helpful whether you are currently doing a qualitative analysis project or considering one, as well as many applications in systematic reviews or literature reviews, or working with mixed-method survey data. Hope to see some of you there, and let us know if you would like us to come to your home town!

It’s quick and easy to set up an on-line survey to collect feedback or research data in a digital format that means you can quickly get straight to analysing the data. Unfortunately, most packages like SurveyMonkey, SurveyGizmo and Kwiksurveys, while all compatible with Quirkos, require a paying subscription before you can actually export any of your data and analyse it.

However, there are two great free platforms we recommend that allow you to run a mixed-method survey, and easily bring all your data into Quirkos to explore and analyse. In this article, we'll go through a step by step guide to setting up a survey in eSurv, and exporting the data to Quirkos

eSurv.org

This is a completely free platform, funded by contributions from Universities, but is available for any use. There are no locked features, or restrictions on responses, and it has an easy to use on-line survey design. There are customisable templates, and you can have custom exit pages too.

Once you have signed up for an account, you will be presented with the screen above, and will be able to get going with your first survey. The first page allows you to name the survey, and set up the title and page description, all have options for changing the text formatting. Just make sure you click on the verification link in the e-mail sent to you, which will allow you to access all the features.

The next screen shows a series of templates you can use to set the style of your survey. Choose one that you like the look of, and you have the option of customising it further with your logo or other colour schemes. Click next.

Now you are ready to start adding questions.

The options box on the right shows all the different types of questions available, and each one has many customisation options at the bottom of the screen. For example, the single text box option can be made to accept only numerical answers, and you can change the maximum length and display size of the box. All questions can be made mandatory, with a custom 'warning' if someone does not fill in that dialogue.

The drag and drop ranking feature is a nice option, and pretty much all the multiple-choice and closed question formats you might want are represented.

When you have chosen the title and settings for each question, you can click on the 'Save & Add Next' button on the top right to quickly add a series of questions, or 'Save & Close' if you are done.

There are also Logic options to add certain questions only in response to certain answers (for example, Please tell us why you didn't like this product). It is of course possible to edit the questions and rearrange them using the drag icon in the main questionnaire overview.

You can test the survey to see how it looks, and when happy click the launch button to make it available to respondents. This also gives you a QR code linking to the survey, allowing smartphone users to complete the survey from a link on posters or printed documents. While you can customise the link title, the web address is always in the format of "https://eSurv.org?u=survey_title".

You can have a large number of surveys on the go at once, and manage them all from the 'Home' screen, which also shows you how many responses you have had.

Once you are ready to analyse your data, open the survey and click on the export button. This gives the options above to select which questions and respondents you want to export, and a date range (useful if you only want to put in new responses). For best use in Quirkos, select the Compact and .csv File format options, and then click download.

The only step you probably want to take before bringing the data into Quirkos is to remove the first row (highlighted above). By default eSurv creates a row which numbers the questions, but it’s usually easier to have the questions themselves as the title, not just the number. Just delete the first row starting with ‘Question’ and this will remove the question numbers, and Quirkos will see the first row with the actual question names. Just save any changes in Excel/LibreOffice making sure you save using the CSV (Comma delimited) format, and ignore the warning that ‘some features may be lost’ and choose ‘Yes’ to keep using that format. You can also remove any columns here that you don’t want (for example e-mail address if it was not provided) but you can also do this in Quirkos.

In Quirkos, start a new Structured Questions project, and select the Import from CSV option from the bottom right 'Add Source' (+) button. Select the file you saved in the previous step, and you will get a preview of the data looking like the screenshot above. Here you have the option to choose which question you want to use for the Source Title (say a name, or respondent ID) and any you might want to ignore, such as IP address. Then make sure that open ended questions are selected as Question, and Property is associated with any discrete or numerical categories. Click import, and voilà!

Should you get new responses, you can add them in the same way to an existing project with the same structure, just make sure when exporting from eSurv that you select the newest responses to export, and don't duplicate older ones.

Now you can use Quirkos to go through and code any of the qualitative text elements, while using the properties and quantitative data  to compare respondents and generate summaries. So for example, you can see the comments that people with negative ratings made side by side by comments from positive feedback, or compare people from different age ranges.

If you need even more customisation of your survey, the open-source platform LimeSurvey, while not as easy to use as eSurv, gives you a vast array of customisability options. LimeService.com allows 25 responses a month for free, but we have our own unrestricted installation available free of charge to our customers – just ask if you need it!

p.s  I've also done a video tutorial covering setting up and using eSurv, and exporting the results into Quirkos.

Later today we are releasing a small update for Quirkos, which adds an important feature users have been requesting: the ability to quickly bring in quantitative and qualitative data from any spreadsheet, or online survey tool such as SurveyMonkey or LimeSurvey.

Now, users can bring in mixed-method data in one click, with the ability to analyse and compare qualitative and quantitative data together. If you have a survey with discrete and quantitative data (such as age, location, or Likert scales) you can use them to stratify and compare open-ended qualitative answers (the Any other comments? Or How can we improve this service?).

Not only will this make bringing data into Quirkos a lot quicker, it will provide a neat workflow for people wanting to understand the qualitative aspects of their data. Now they can code and develop frameworks to understand comments and written data sources, which may hold the key to understanding something important that isn’t shown in the quantitative data.

In Quirkos, this functionality is provided as a new option in the ‘Add Source’ button on the bottom left of a project. Users should create a new ‘Structured Question’ project, which gives the same Questions as sections in the qualitative text of the source. The discrete and quantitative data will be imported as source properties which describe each response in the survey.

To bring spreadsheet or tabulated data into Quirkos, you need to have it in CSV format (comma separated variables) which is a standard file format that most platforms can use to export data. If that format is not supported by your data collection workflow, as long as it can be imported into Excel or another spreadsheet package such as Google Docs or LibreOffice Calc. All these packages allow you to save a table of data in CSV format, and you should select the default comma, not tab separated format. The first row, should be the titles you want the properties and questions to be.

Quirkos will try and automatically guess which columns represent discrete properties (such as name or age) and which ones are sentences. It does this in a simple way: any row titles with a question, such as “How did you feel about this event?” will become a long-text qualitative question and answer, or if the answer contains spaces like a sentence structure. Otherwise, it will suggest import as a source property for a value like age or name. If this does not come through as you wish, there is the drop-down option to change how that row is imported.

This provides 4 options. Source Title is the name you wish to give each source in the project. This might be a name, or a ID number – and you can only select one property to be the source title. Property is for source properties, the quantitative or discrete data that describes the source. Question is for the open ended qualitative text sections, and finally there is an option for ‘Ignore’ if there was a field or value you did not want to bring into the project.

It is possible to keep adding more and more sources in this way, for example if you had later additions to a survey. However, it will also create duplicates of data already in the project (in case something changed) so make sure that a new CSV file being imported doesn’t contain the old responses.

If you already have Quirkos, all you need to do is download the new installer for version 1.2 for Windows or Mac, and follow the install procedure. This will install the new version over the old one (v1.1) and there will be no changes to your shortcuts, projects or license. The update is free for everyone, even if you are using the free-trial, and once again, there are no compatibility problems with older project files.

Evaluating programmes and projects are an essential part of the feedback loop that should lead to better services. In fact, programmes should be designed with evaluations in mind, to make sure that there are defined and measurable outcomes.

While most evaluations generally include numerical analysis, qualitative data is often used along-side the quantitative, to show richness of project impact, and put a human voice in the process. Especially when a project doesn’t meet targets, or have the desired level of impact, comments from project managers and service users usually give the most information into what went wrong (or right) and why.

For smaller pilot and feasibility projects, qualitative data is often the mainstay of the evaluation data, when numerical data wouldn’t reach statistical analysis, or when it is too early in a programme to measure intended impact. For example, a programme looking at obesity reductions might not be able to demonstrate a lower number of diabetes referrals at first, but qualitative insight in the first year or few months of the project might show how well messages from the project are being received, or if targeted groups are talking about changing their behaviour. When goals like this are long term (and in public health and community interventions they often are) it’s important to continuously assess the precursors to impact: namely engagement, and this is usually best done in a qualitative way.

So, what is best practice for qualitative evaluations? Fortunately, there are some really good guides and overviews that can help teams choose the right qualitative approach. Vaterlaus and Higgenbotham give a great overview of qualitative evaluation methods, while Professor Frank Vanclay talks at a wider level about qualitative evaluations, and innovative ways to capture stories. However, there was also a nice ‘tick-box’ style guide produced by the old Public Health Resource Unit which can still be found at this link. Essentially, the tool suggests 10 questions that can be used to assess the quality of a qualitative based-evaluation – really useful when looking at evaluations that come from other fields or departments.

But my contention is that the appraisal tool above is best implemented as a guide for producing qualitative evaluations. If you start by considering the best approach, how you are going to demonstrate rigour, choosing appropriate methods and recruitment, you’ll get a better report at the end of it. I’d like to discuss and expand on some of the questions used to assess the rigour of the qualitative work, because this is something that often worries people about qualitative research, and these steps help demystify good practice.

1. The process: Start by planning the whole evaluation from the outset: What do you plan to do? All the rest will then fall into place.
    
2. The research questions: what are they and why were these chosen? Are the questions going to give the evaluation the data it needs, and will the methods capture that correctly?
    
3. Recruitment: who did you choose, and why? Who didn’t take part, and how did you find people? What gaps are there likely to be in representing the target group, and how can you compensate for this? Were there any ethical considerations, how was consent gained, and what was the relationship between the participants and the researcher(s)? Did they have any reason to be biased or not truthful?
    
4. The data: how did you know that enough had been collected? (Usually when you are starting to hear the same things over and over – saturation) How was it recorded, transcribed, and was it of good quality? Were people willing to give detailed answers?
    
5. Analysis: make sure you describe how it was done, and what techniques were used (such as discourse or thematic analysis). How does the report choose which quotes to reproduce, and are there contradictions reported in the data? What was the role of the researcher – should they declare a bias, and were multiple views sought in the interpretation of the data?
    
6. Findings: do they meet the aims and research questions? If not, what needs to be done next time? Are there clear findings and action points, appropriate to improving the project?

Then the final step for me is the most important of all: SHARE! Don't let it end up on a dusty shelf! Evaluations are usually seen as a tedious but necessary internal process, but they can be so useful to people as case-studies and learning tools in organisations and groups you might never have thought of. This is especially true if there are things that went wrong, help someone in another local authority not make the same mistakes!

At the moment the best UK repositories of evaluations are based around health and economic benefits but that doesn’t stop you putting the report on your organisation’s website – if someone is looking for a similar project, search engines will do the leg work for you. That evaluation might save someone a lot of time and money, and it goes without saying, look for any similar work before you start a project, you might get some good ideas, and stop yourself falling into the same pit-holes!

We've now put up the summary report for our qualitative research project on the Scottish Referendum, which we analysed using Quirkos. You can download the PDF of the 10 page report from the link above, I hope you find something interesting in there! The full title is "Overview of a qualitative study on the impact of the 2014 referendum for Scottish independence in Edinburgh, and views of the political process" and here's the summary findings:

"The interviews revealed a great depth of understanding of a wide range of political issues, and a nuanced understanding of many arguments for and against independence. Many people described some uncertainty about which way to vote, but it did not seem that anyone had changed their mind over the course of the campaigning.

There was a general negative opinion towards the general political system, especially Westminster, from both yes and no voters. Participants had varying opinions on political leaders and parties, even though some people were active members of political parties. Yes and No supporters both felt that the No campaign was poorly run, and used too many negative messages, this feeling was especially strong in No voters.
The most important concerns for responders was about public finances, financial stability of an independent Scotland, the issue of currency for Scotland was often mentioned, but often with distrust of politicians comments on the subject. Westminster induced austerity and the future of the NHS also featured as important policy considerations.

People expressed generally negative views of the media portrayal of the referendum, most feeling that newspapers and especially the BBC had been biased, although No supporters were more likely to find the media balanced.

In general, people felt that the process had been good for Scotland, even No supporters, and there was general support for greater devolution of powers. People had seen the process as being very positive for the SNP, and nearly all respondents felt the Yes campaign had been well run. People expressed a negative view of the Labour party during the campaign, although voters also mentioned strong criticism of Labour’s wider policy position in recent years. People had generally positive opinions of Nicola Sturgeon, mixed reactions to Alex Salmond, and generally negative comments on Ed Miliband’s public image, while also stating that this should not be an important factor for voters. People believed that the polls would be correct in predicting a swing from Labour to the SNP in Scotland.

Many expressed a belief that the level of debate in Edinburgh had been good, and that the Yes campaign was very visible. Respondents were positive about the inclusion of voters from the age of 16, were surprised at how much support the Yes campaign generated, and some felt that a future referendum would be successful in gaining independence for Scotland."

The report also contains some information about the coding process using Quirkos:

"The interviews together lasted 6.5 hours and once transcribed comprised just under 58000 words, an average of 4800 words per interview. 75 themes were used to code the project, with 3160 coding events logged, although each text may cover multiple coding events. In total, 87% of the text was coded with at least one topic. The coding took an experienced coder approximately 7 hours (over a three day period) once any breaks longer than 5 minutes were removed, an average of one code every 8 seconds."

Personally, I've been really happy doing this project with Quirkos, and especially with how quick it took to do the coding. Obviously, with any qualitative analysis process there is a lot of reading, thinking and mind-changing that happens from setting the research questions to writing up a report. However, I really do think that Quirkos makes the coding and exploration process quicker, and I do love how much one can play with the data, just looking to see how much keywords come up, or whether there are connections between certain themes.

In this project, the cluster views (one for media bias shown above) were really revealing, and sometimes surprising. But the side-by-side queries were also really useful for looking to see differences in opinions between Yes and No supporters, and also to demonstrate there was little difference in the quotes from men and women – they seemed to largely care about the same issues, and used similar language.

Feel free to see for yourself though, all the transcripts, as well as the coded project file can be downloaded from our workshop materials pages, so do let me know if Quirkos lets you have a different view on the data!

If the polls are to be believed, Labour are going to loose a lot of Scottish seats in Westminster to the SNP next month. This wave of support seems to come largely out of the referendum last year on Scottish Independence, but it's difficult to completely understand why this is. Qualitative research to the rescue!

“I mean, it used to just be like, “If you dislike the Tories, you had to vote Labour,” and then you kind of vote Lib Dem but now the SNP are a realistic choice as well.”

Earlier this year we ourselves conducted a fun little study looking at what people thought about the campaigning for the Referendum, and how this affected views on political parties and the prospects for the 2015 general election. We interviewed 12 people, face to face, all voters living in Edinburgh. It wasn’t intended to be a representative sample, but does capture a range of ages and political leanings.

We did this mostly to demonstrate how Quirkos can be used to understand complex social issues, and help expert researchers understand qualitative data. We also make all of the transcripts of the project available to download on our website for use in training workshops.

This week we are releasing a couple of insights into the data, created and coded with Quirkos. Personally, I was struck by how nuanced and detailed people’s political understanding was, and how far removed it seems to be from the mainstream media’s summations of major political messages.

When we look at how people describe Labour and the SNP (using the overlapping codes function) there are some interesting insights:

“I think they were just in a bit of a muddle because a lot of the SNP’s policies are quite left wing, I think, and Labour ought to be, that should be their ground so the SNP were sort of stealing Labour’s voters. Not stealing in a bad way but taking over Labour ground, so the Labour party have lost out on support because they’ve tried to go a bit more middle to counteract the Conservatives.”

“it’s sometimes harder to distinguish the Labour policies from the Conservative policies and I think that was under Tony Blair that drift to the right happened so I can understand why that has been a vacuum created and the SNP has managed to fill it. I have to say, I used to think of the SNP as a right wing party and they’re not. They’re a very left wing party, aren’t they?”

But it is clear that while Labour supporting the ‘Better Together’ campaign had a negative impact for some, wider policy issues seem to be the main reason, as shown when looking at overlapping codes for Negative and Labour:

“a lot of Scottish people used to really like the Labour party and I think they haven’t done themselves any favours in the referendum, you know. Ed Miliband coming up at the last gasp but not actually knowing where the Fife was and things like that”

“I mean, a big one in UK terms is this whole programme of austerity and how it’s been dealt with. Perhaps not so many people have seen through that now, but anyone who really probes the policies, you’ll find that the Labour party’s policy isn’t that much different, even in a matter of degrees or billions of pounds or whatever it is, but actually fundamentally, they’re still supporting austerity as well”

“There is enough power in Scotland to move towards the left. There is party that could do something so why not vote for them? So, I think that’s how my thinking goes and I think it’s a lot of people. I mean, I thinks that’s why Labour is now in real trouble because they’ve identified with what they did in power for 14 years or however long it was.”

While others note that it is a difficult balancing act:

“I think they’re kind of playing both sides, you know, they’re trying to promote that they’re a Scottish party looking after the interests of Scotland, but we’re also part of a bigger organisation that’s looking at the needs of the UK as a whole or a particular area.”

You can download a more detailed list of quotes from here, and don’t forget that the full transcripts are also available from our workshop page. More insights to come in the next few days!

When doing analysis and coding in a qualitative research project, it is easy to become completely focused on the thematic framework, and deciding what a section of text is about. However, qualitative analysis software is a useful tool for organising more than just the topics in the text, they can also be used for deeper contextual and meta-level analysis of the coding and data.

Because you can pretty much record and categorise anything you can think of, and assign multiple codes to one section of text, it often helps to have codes about the analysis that help with managing quotes later, and assisting in deeper conceptual issues. So some coders use some sort of ranking system so they can find the best quotes quickly. Or you can have a category for quotes that challenge your research questions, or seem to contradict other sources or findings. Here are 6 suggestions for these meta-level codes you could create in your qualitative project (be it Quirkos, Nvivo, Atlas-ti or anything!):

Rating
I always have a node I call ‘Key Quotes’ where I keep track of the best verbatim snippets from the text or interview. It’s for the excited feeling you get when someone you interviewed sums up a problem or your research question in exactly the right way, and you know that you are going to end up using that quote in an article. Or even for the title of the article!

However, another way you can manage quotes is to give them a ranking scheme. This was suggested to me by a PhD student at Edinburgh, who gives quotes a ranking from 1-5, with each ‘star-rating’ as a separate code. That way, it’s easy to cross reference, and find all the best quotes on a particular topic. If there aren’t any 5* quotes, you can work down to look at the 4 star, or 3 star quotes. It’s a quick way to find the ‘best’ content, or show who is saying the best stuff. Obviously, you can do this with as little or much detail as you like, ranking from 1-10 or just having ‘Good’ and ‘Bad’ quotes.

Now, this might sound like a laborious process, effectively adding another layer of coding. However, once you are in the habit, it really takes very little extra time and can make writing up a lot quicker (especially with large projects). By using the keyboard shortcuts in Quirkos, it will only take a second more. Just assign the keyboard numbers 1-5 to the appropriate ranking code, and because Quirkos keeps the highlighted section of text active after coding, you can quickly add to multiple categories. Drag and drop onto your themes, and hit a number on the keyboard to rank it. Done!

Contradictions
It is sometimes useful to record in one place the contradictions in the project – this might be within the source, where one person contradicts themselves, or if a statement contradicts something said by another respondent. You could even have a separate code for each type of contradiction. Keeping track of these can not only help you see difficult sections of data you might want to review again, but also show when people are being unsure or even deceptive in their answers on a difficult subject. The overlap view in Quirkos could quickly show you what topics people were contradicting themselves about – maybe a particular event, or difficult subject, and the query views can show you if particular people were contradicting themselves more than others.

Ambiguities
In qualitative interview data where people are talking in an informal way about their stories and lives, people often say things where the meaning isn’t clear – especially to an external party. By collating ambiguous statements, the researcher has the ability to go back at the end of the source and see if each meaning is any clearer, or just flag quotes that might be useful, but might be at risk of being misinterpreted by the coder.

Not-sures
Slightly different from ambiguities: these are occasions when the meaning is clear enough, but the coder is not 100% sure that it belongs in a particular category. This often happens during a grounded theory process where one category might be too vague and needs to be split into multiple codes, or when a code could be about two different things.

Having a not-sure category can really help the speed of the coding process. Rather than worrying about how to define a section of text, and then having sleepless nights about the accuracy of your coding, tag it as ‘Not sure’ and come back to it at the end. You might have a better idea where they all belong after you have coded some more sources, and you’ll have a record of which topics are unclear. If you are not sure about a large number of quotes assigned to the ‘feelings’ Quirk (again, shown by clustering in the overlap view in Quirkos), you might want to consider breaking them out into an ‘emotions’ and ‘opinions’ category later!

Challenges
I know how tempting it can be to go through qualitative analysis as if it were a tick-box exercise, trying to find quotes that back up the research hypothesis. We’ve talked about reflexivity before in this blog, but it is easy to go through large amounts of data and pick out the bits that fit what you believe or are looking for. I think that a good defence against this tendency is to specifically look for quotes that challenge you, your assumptions or the research questions. Having a Quirk or node that logs all of these challenges lets you make sure you are catching them (and not glossing over them) and secondly provides a way to do a validity assessment at the end of coding: Do these quotes suggest your hypothesis is wrong? Can you find a reason that these quotes or individuals don’t fit your theory? Usually these are the most revealing parts of qualitative research.

Absences
Actually, I don’t know a neat way to capture the essence of something that isn’t in the data, but I think it’s an important consideration in the analysis process. With sensitive topics, it is sometimes clear to the researcher that an important issue is being actively avoided, especially if an answer seems to evade the question. These can be at least coded as absences at the interviewer’s question. However, if people are not discussing something that was expected as part of the research question, or was an issue for some people but not others, it is important to record and acknowledge this. Absence of relevant themes is usually best recorded in memos for that source, rather than trying to code non-existent text!

These are just a few suggestions, if you have any other tips you’d like to share, do send them to daniel@quirkos.com or start a discussion in the forum. As always, good luck with your coding!

We are running more and more workshops helping people learn qualitative analysis and Quirkos. I always feel that the best way to learn is by doing, and the best way to remember is through play. To this end, we have created two sources of qualitative data that anyone can download and use (with any package) to learn how to use software for qualitative data analysis.

These can be found at the workshops folder. There are two different example data sets, which are free for any training use. The first is a basic example project, which is comprised of a set of fictional interviews with people talking about what they generally have for breakfast. This is not really a gripping exposé of a critical social issue, but is short and easy to engage with, and already provides some suprises when it comes to exploring the data. The materials provided include individual transcribed sources of text, in a variety of formats that can be brought into Quirkos. The idea is that users can learn how to bring sources into Quirkos, create a basic coding framework, and get going on coding data.

For the impatient, there is also a 'here's one we created earlier' file, in which all the sources have been added to the project, described age and gender and occupation as source properties, a completed framing codework, and a good amount of coding. This is a good starting point if someone wants to use the various tools to explore coded data and generate outputs. There is also a sample report, demonstrating what a default output looks like when generated by Quirkos, including the 'data' folder, which includes all the pictures for embedding in a report or PowerPoint presentation.

This is the example project we most frequently use in workshops. It allows us to quickly cover all the major steps in qualitative analysis with software, with a fun and easy to understand dataset. It also lets us see some connections in the data, for example how people don't describe coffee as a healthy option, and that women for some reason talk about toast much more than men.

However, the breakfast example is not real qualitative data - it is short, and fictitious, so for people who come along to our more advanced analysis workshops, we are happy to now make available a much more detailed and lively dataset. We have recently completed a project on the impact on voter opinions in Scotland after the 2014 Referendum for independence. This comprises of 12 semi-structured interviews with voters based in Edinburgh, on their views on the referendum process, and how it has changed their outlook on politics and voting in the run-up to the 2015 General Election in the UK.

When we conducted these interviews, we explicitly got consent for them to be made publicly available and used for workshops after they had been transcribed and anonymised. This gives us a much deeper source of data to analyse in workshops, but also allows for anyone to download a rich set of data to use in their own time (again with any qualitative software package) to practice their analytical skills in qualitative research. You can download these interviews and further materials at this link.

We hope you will find these resources useful, please acknowledge their origin (ie Quirkos), let us know if you use them in your training and learning process, and if you have any feedback or suggestions.