The importance of keeping open-ended qualitative responses in surveys

open-ended qualitative responses in surveys

I once had a very interesting conversation at a MRS event with a market researcher from a major media company. He told me that they were increasingly ‘costing-out’ the qualitative open-ended questions from customer surveys because they were too expensive and time consuming to analyse. Increasingly they were replacing open-ended questions with a series of Likert scale questions which could be automatically and statistically examined.

 

I hear similar arguments a lot, and I totally understand the sentiment: doing good qualitative research is expensive, and requires good interpretation. However, it’s just as possible to do statistical analysis poorly, and come up with meaningless and inaccurate answers. For example, when working with Likert scales, you have to be careful about which parametric tests you use, and make sure that the data is normally distributed (Sullivan and Artino 2013).

 

There is evidence that increasing the number of options in closed questions does not significantly change the responses participants share (Dawes 2008), so if you need a good level of nuance into customer perceptions, why not let your users choose their own words. “Quick Qual” approaches, like asking people to use one word to describe the product or their experience can be really illuminating. Better yet, these responses are easy to analyse, and present as an engaging word cloud!

 

Even when you have longer responses, it’s not necessary to always take a full classification and quantification approach to qualitative survey data such as in Nardo (2003). For most market research investigations, this level of detail is not needed by researcher or client.

 

Indeed, you don’t need to do deep analysis of the data to get some value from it. A quick read through some of the comments can make sure your questions are on track, and there aren’t other common issues being raised. It helps check you were asking the right questions, and can help explain why answers for some people aren’t matching up with the rest. As ever, qualitative data is great for surprises, responses you hadn’t thought of, and understanding motivations.

 

Removing open ended questions means you can’t provide nice quotes or verbatims from the feedback, which are great for grounding a report and making it come to life. If you have no quotes from respondents, you also are missing the opportunity to create marketing campaigns around comments from customer evangelists, something Lidl UK has done well by featuring positive Tweets about their brand. In this article marketing director Claire Farrant notes the importance of listening and engaging with customer feedback in this way. It can also make people more satisfied with the feedback process if they have a chance to voice their opinions in more depth.

 

I think it’s also vital to include open-ended questions when piloting a survey or questionnaire. Having qualitative data at an early stage can let you refine your questions, and the possible responses. Sometimes the language used by respondents is important to reflect when setting closed questions: you don’t want to be asking questions like “How practical did you find this product” when the most common term coming from the qualitative data is “Durable”. It’s not always necessary to capture and analyse qualitative data for thousands of responses, but looking at a sample of a few dozen or hundred can show if you are on the right track before a big push.

 

You also shouldn’t worry too much about open-ended surveys having lower completion rates. A huge study by SurveyMonkey found that a single open question actually increased engagement slightly, and only when there were 5 or more open-ended response boxes did this have a negative impact on completion.

 

Finally, without qualitative responses, you lose the ability to triangulate and integrate your qualitative and quantitative data: one of the most powerful tools in survey analysis. For example, in Quirkos it is trivial to do very quick comparative subset analysis, using any of the closed questions as a pivot point. So you can look at the open ended responses from people who gave high satisfaction scores next to those that were low, and rather than then being stuck trying to explain the difference in opinion, you can look at the written comments to get an insight into why they differ.

 

And I think this is key to creating good reports for clients. Usually, the end point for a customer is not being told that 83% of their customers are satisfied with their helpline: they want to actions that will improve or optimise delivery. What exactly was the reason 17% of people had a bad experience? It’s all very well to create an elaborate chain of closed questions, such as ‘You said you were unsatisfied. Which of these reasons bests explains this? You said the response time made you unsatisfied. How long did you wait? 0-3min, 3-5min etc. etc. But these types of surveys are time consuming to program and make comprehensive, and sometimes just allowing someone to type “I had to wait more than 15 minutes for a response” would have given you all the data you needed on a critical point.

 

The depth and insight from qualitative data can illuminate differences in respondent’s experiences, and give the key information to move things forward. Instead of thinking how can you cost-out qualitative responses, think instead how you can make sure they are integrated to provide maximum client value! A partnership between closed and open questions is usually the most powerful way to get both a quick summary and deep insight into complex interactions, and there is no need to be afraid of the open box!

 

Quirkos is designed to make it easy to bring both qualitative and quantitative data from surveys together, and use the intuitive visual interface to explore and play with market research data. Download a free trial of our qualitative analysis software, or contact us for a demo, and see how quickly you can step-up from paper based analysis into a streamlined and insightful MRX workflow!

 

How to set up a free online mixed methods survey

It’s quick and easy to set up an on-line survey to collect feedback or research data in a digital format that means you can quickly get straight to analysing the data. Unfortunately, most packages like SurveyMonkey, SurveyGizmo and Kwiksurveys, while all compatible with Quirkos, require a paying subscription before you can actually export any of your data and analyse it.

 

However, there are two great free platforms we recommend that allow you to run a mixed-method survey, and easily bring all your data into Quirkos to explore and analyse. In this article, we'll go through a step by step guide to setting up a survey in eSurv, and exporting the data to Quirkos

 

eSurv.org

This is a completely free platform, funded by contributions from Universities, but is available for any use. There are no locked features, or restrictions on responses, and it has an easy to use on-line survey design. There are customisable templates, and you can have custom exit pages too.

Once you have signed up for an account, you will be presented with the screen above, and will be able to get going with your first survey. The first page allows you to name the survey, and set up the title and page description, all have options for changing the text formatting. Just make sure you click on the verification link in the e-mail sent to you, which will allow you to access all the features.

 

The next screen shows a series of templates you can use to set the style of your survey. Choose one that you like the look of, and you have the option of customising it further with your logo or other colour schemes. Click next.

Now you are ready to start adding questions.

 

The options box on the right shows all the different types of questions available, and each one has many customisation options at the bottom of the screen. For example, the single text box option can be made to accept only numerical answers, and you can change the maximum length and display size of the box. All questions can be made mandatory, with a custom 'warning' if someone does not fill in that dialogue.

 

The drag and drop ranking feature is a nice option, and pretty much all the multiple-choice and closed question formats you might want are represented.

 

When you have chosen the title and settings for each question, you can click on the 'Save & Add Next' button on the top right to quickly add a series of questions, or 'Save & Close' if you are done.

 

There are also Logic options to add certain questions only in response to certain answers (for example, Please tell us why you didn't like this product). It is of course possible to edit the questions and rearrange them using the drag icon in the main questionnaire overview.

 

You can test the survey to see how it looks, and when happy click the launch button to make it available to respondents. This also gives you a QR code linking to the survey, allowing smartphone users to complete the survey from a link on posters or printed documents. While you can customise the link title, the web address is always in the format of "https://eSurv.org?u=survey_title".

 

You can have a large number of surveys on the go at once, and manage them all from the 'Home' screen, which also shows you how many responses you have had.

 

Once you are ready to analyse your data, open the survey and click on the export button. This gives the options above to select which questions and respondents you want to export, and a date range (useful if you only want to put in new responses). For best use in Quirkos, select the Compact and .csv File format options, and then click download.

 

exported csv file in excel

The only step you probably want to take before bringing the data into Quirkos is to remove the first row (highlighted above). By default eSurv creates a row which numbers the questions, but it’s usually easier to have the questions themselves as the title, not just the number. Just delete the first row starting with ‘Question’ and this will remove the question numbers, and Quirkos will see the first row with the actual question names. Just save any changes in Excel/LibreOffice making sure you save using the CSV (Comma delimited) format, and ignore the warning that ‘some features may be lost’ and choose ‘Yes’ to keep using that format. You can also remove any columns here that you don’t want (for example e-mail address if it was not provided) but you can also do this in Quirkos.

 

In Quirkos, start a new Structured Questions project, and select the Import from CSV option from the bottom right 'Add Source' (+) button. Select the file you saved in the previous step, and you will get a preview of the data looking like the screenshot above. Here you have the option to choose which question you want to use for the Source Title (say a name, or respondent ID) and any you might want to ignore, such as IP address. Then make sure that open ended questions are selected as Question, and Property is associated with any discrete or numerical categories. Click import, and voilà!

 

Should you get new responses, you can add them in the same way to an existing project with the same structure, just make sure when exporting from eSurv that you select the newest responses to export, and don't duplicate older ones.

 

Now you can use Quirkos to go through and code any of the qualitative text elements, while using the properties and quantitative data  to compare respondents and generate summaries. So for example, you can see the comments that people with negative ratings made side by side by comments from positive feedback, or compare people from different age ranges.

 

If you need even more customisation of your survey, the open-source platform LimeSurvey, while not as easy to use as eSurv, gives you a vast array of customisability options. LimeService.com allows 25 responses a month for free, but we have our own unrestricted installation available free of charge to our customers – just ask if you need it!

 

p.s  I've also done a video tutorial covering setting up and using eSurv, and exporting the results into Quirkos.

Bringing survey data and mixed-method research into Quirkos

quirkos spreadsheet

 

Later today we are releasing a small update for Quirkos, which adds an important feature users have been requesting: the ability to quickly bring in quantitative and qualitative data from any spreadsheet, or online survey tool such as SurveyMonkey or LimeSurvey.

 

Now, users can bring in mixed-method data in one click, with the ability to analyse and compare qualitative and quantitative data together. If you have a survey with discrete and quantitative data (such as age, location, or Likert scales) you can use them to stratify and compare open-ended qualitative answers (the Any other comments? Or How can we improve this service?).

 

Not only will this make bringing data into Quirkos a lot quicker, it will provide a neat workflow for people wanting to understand the qualitative aspects of their data. Now they can code and develop frameworks to understand comments and written data sources, which may hold the key to understanding something important that isn’t shown in the quantitative data.

 

import csv dialogue quirkos

In Quirkos, this functionality is provided as a new option in the ‘Add Source’ button on the bottom left of a project. Users should create a new ‘Structured Question’ project, which gives the same Questions as sections in the qualitative text of the source. The discrete and quantitative data will be imported as source properties which describe each response in the survey.

 

To bring spreadsheet or tabulated data into Quirkos, you need to have it in CSV format (comma separated variables) which is a standard file format that most platforms can use to export data. If that format is not supported by your data collection workflow, as long as it can be imported into Excel or another spreadsheet package such as Google Docs or LibreOffice Calc. All these packages allow you to save a table of data in CSV format, and you should select the default comma, not tab separated format. The first row, should be the titles you want the properties and questions to be.

 

Quirkos will try and automatically guess which columns represent discrete properties (such as name or age) and which ones are sentences. It does this in a simple way: any row titles with a question, such as “How did you feel about this event?” will become a long-text qualitative question and answer, or if the answer contains spaces like a sentence structure. Otherwise, it will suggest import as a source property for a value like age or name. If this does not come through as you wish, there is the drop-down option to change how that row is imported.

 

This provides 4 options. Source Title is the name you wish to give each source in the project. This might be a name, or a ID number – and you can only select one property to be the source title. Property is for source properties, the quantitative or discrete data that describes the source. Question is for the open ended qualitative text sections, and finally there is an option for ‘Ignore’ if there was a field or value you did not want to bring into the project.


It is possible to keep adding more and more sources in this way, for example if you had later additions to a survey. However, it will also create duplicates of data already in the project (in case something changed) so make sure that a new CSV file being imported doesn’t contain the old responses.

 

If you already have Quirkos, all you need to do is download the new installer for version 1.2 for Windows or Mac, and follow the install procedure. This will install the new version over the old one (v1.1) and there will be no changes to your shortcuts, projects or license. The update is free for everyone, even if you are using the free-trial, and once again, there are no compatibility problems with older project files.