Free materials for qualitative workshops

qualitative workshop on laptops with quirkos

 

We are running more and more workshops helping people learn qualitative analysis and Quirkos. I always feel that the best way to learn is by doing, and the best way to remember is through play. To this end, we have created two sources of qualitative data that anyone can download and use (with any package) to learn how to use software for qualitative data analysis.

 

These can be found at the workshops folder. There are two different example data sets, which are free for any training use. The first is a basic example project, which is comprised of a set of fictional interviews with people talking about what they generally have for breakfast. This is not really a gripping exposé of a critical social issue, but is short and easy to engage with, and already provides some suprises when it comes to exploring the data. The materials provided include individual transcribed sources of text, in a variety of formats that can be brought into Quirkos. The idea is that users can learn how to bring sources into Quirkos, create a basic coding framework, and get going on coding data.


For the impatient, there is also a 'here's one we created earlier' file, in which all the sources have been added to the project, described age and gender and occupation as source properties, a completed framing codework, and a good amount of coding. This is a good starting point if someone wants to use the various tools to explore coded data and generate outputs. There is also a sample report, demonstrating what a default output looks like when generated by Quirkos, including the 'data' folder, which includes all the pictures for embedding in a report or PowerPoint presentation.

 

This is the example project we most frequently use in workshops. It allows us to quickly cover all the major steps in qualitative analysis with software, with a fun and easy to understand dataset. It also lets us see some connections in the data, for example how people don't describe coffee as a healthy option, and that women for some reason talk about toast much more than men.

 

However, the breakfast example is not real qualitative data - it is short, and fictitious, so for people who come along to our more advanced analysis workshops, we are happy to now make available a much more detailed and lively dataset. We have recently completed a project on the impact on voter opinions in Scotland after the 2014 Referendum for independence. This comprises of 12 semi-structured interviews with voters based in Edinburgh, on their views on the referendum process, and how it has changed their outlook on politics and voting in the run-up to the 2015 General Election in the UK.

 

When we conducted these interviews, we explicitly got consent for them to be made publicly available and used for workshops after they had been transcribed and anonymised. This gives us a much deeper source of data to analyse in workshops, but also allows for anyone to download a rich set of data to use in their own time (again with any qualitative software package) to practice their analytical skills in qualitative research. You can download these interviews and further materials at this link.

 

We hope you will find these resources useful, please acknowledge their origin (ie Quirkos), let us know if you use them in your training and learning process, and if you have any feedback or suggestions.

Qualitative data in the UK Public Sector

queuing for health services

 

The last research project I worked on with the NIHR was a close collaboration between several universities, local authorities and NHS trusts. We were looking at evidence use by managers in the NHS, and one of the common stories we heard was how valuable information often ended up on the shelf, and not used to inform service provision or policy.


It was always a real challenge for local groups, researchers and academics to create research outputs that were in a digestible format so that they could be used by decision makers, who often had very short timescales and limited resources. We also were told of the importance of using examples and case studies of other trusts or departments that had successes: it’s all very well making suggestions to improve services, but most of the time, the battle is getting that into practice. It’s one of the reasons we created a mini-case study guide, short one page summaries of ‘best-practice’ – places where a new approach had worked and made changes.


However, the biggest shock for me was how difficult it was to engage qualitative data in decision making. In many public sector organisations, qualitative data is seen as the poor-cousin of quantitative statistics, only used when figures can’t be found, or the interest group is too small for statistical significant findings.


So many wonderful sources of qualitative data seemed to be sitting around, collecting dust: including research from community organisations, consultations, and feedback from service users – data that had already been collected, and was awaiting analysis for a specific purpose. There was also a lack of confidence in some researchers in how to work with qualitative data, and an understandable sense that it was a very time consuming process. At best, qualitative data was just providing quotes to illustrate reports like JSNAs which were mostly filled with quantitative data.

 

A big part of the problem seemed to be how decision makers, especially from a clinical background, were more comfortable with quantitative data. For managers used to dealing with financial information, RCTs, efficacy trials etc., this is again quite understandable, and they were used to seeing graphs and tests of statistical significance. But there was a real chicken-and-egg problem: because they rarely took into account qualitative data, it was rarely requested, and there was little incentive to improve qualitative analytical skills.


One group we spoke to had produced a lovely report on a health intervention for an ethnic minority group. Their in-depth qualitative interviews and focus groups had revealed exactly why the usual health promotion message wasn’t getting though, and a better approach to engage with this population. However, the first time they went to present their findings to a funding board, the members were confused. The presentation was too long, had too many words, and no graphs. As one of many items on the agenda, they had to make a case in five minutes and a few slides.


So that’s just what they did. They turned all their qualitative data into a few graphs, which supported their case for an intervention in this group. Personally, it was heart-breaking to see all this rich data end up on the cutting-room floor, but evidence is not useful unless it is acted upon. Besides, the knowledge that the team had from this research meant that with their budget now approved, they knew they could craft the right messages for an effective campaign.


This story was often in my mind when we were designing Quirkos – what would the outputs look like that would have an impact on decision makers? It had to produce visual summaries, graphs and quotes that can be put into a PowerPoint presentation. And why couldn’t the interface itself be used to present the data? If the audience asked a question about a particular quote or group, couldn’t the presenter show that to them there and then?

 

Opening the door to make qualitative data easier to work with and visualise is one thing, but a whole culture of change is needed in many organisations to improve the understanding and use of qualitative data. Until this happens, many evidence based decisions are only being made on the basis of a limited style and depth of data, and valuable insights are being missed.

 

With the prospect of sustained and continued cuts to public services in the UK, there are fewer chances to get something right. Qualitative engagement here can tell us not only what needs to be done and how to learn from our mistakes, but how to get it right the first time.

 



 

Upgrade from paper with Quirkos

qualitative analysis with paper

Having been round many market research firms in the last few months, the most striking things is the piles of paper, or at least in the neater offices - shelves of paper!

When we talk to small market research firms about their analysis process, many are doing most of their research by printing out data and transcripts, and coding them with coloured highlighters. Some are adamant that this is the way that works best for them, but others are a little embarrassed at the way they are still using so much time and paper with physical methods.

 

The challenge is clear – the short turn-around time demanded by clients doesn't leave much time for experimenting with new ways of working, and the few we had talked to who had tried qualitative analysis software quickly felt this wasn't something they were able to pick up quickly.

 

So, most of the small Market Research agencies with less than 5 associates (as many as 75% of firms in the UK) are still relying on work-flows that are difficult to share, don't allow for searching across work, and don't have an undo button! Not to mention the ecological impact of all that printing, and the risk to deadlines from an ill placed mug of coffee.

 

That's one of the reasons we created Quirkos, and why we are launching our new campaign this week at the Market Research Society annual conference in London. Just go to our new website, www.upgradefrompaper.com and watch our fun, one minute video about drowning in paper, and how Quirkos can help.

Quirkos isn't like other software, it is designed to mimic the physical action of highlighting and coding text on paper with an intuitive interface that you can use to get coding right away. In fact, we bet you can get coding a project before your printer has got the first source out of the tray.

 

You no longer need days of training to use qualitative analysis software, and Quirkos has all the advantages you'd expect, such as quick searches, full undo-redo capability and lots of flexibility to rearrange your data and framework. But it also has other pleasant surprises: there's no save button, because work is automatically saved after each action. And it creates graphical reports you can share with colleagues or clients.

 

Finally, you can export your work at any stage to Word, and print it out (if you so wish!) with all your coding and annotations as familiar coloured highlights – ideal to share, or just to help ease the transition to digital. It's always comforting to know you can go back to old habits at any time, and not loose the work you've already done!

 

It's obviously not just for market research firms; students, academics and charities who have either not tried any qualitative software before, or found the other options too confusing or expensive can reduce their carbon footprint and save on their department's printing costs!

 

So take the leap, and try it out for a month, completely free, on us. Upgrade from paper to Quirkos, and get a clear picture of your research!

 

www.upgradefrompaper.com


p.s. All the drawings in our video were done by our very own Kristin Schroeder! Not bad, eh?

Quirkos v1.1 is here!

We are excited to announce that the first update for Quirkos can now be downloaded from here!

 

Version 1.1 adds two main new features: batch import, and mutli-language reports.

 

If you have a large number of text sources or transcripts to add to a project, you can now do it all in one go, without having to import each seperately. Just click on the (+) add source button on the bottom right of the source view, and select Import Multiple Sources. You have the option to select certain files, or a whole folder at once. This makes it a lot quicker to get going with Quirkos from an existing dataset.

 

Secondly, we've improved the export options (including HTML, PDF, CSV and Word files) so that they all support projects with multiple languages and non-latin scripts. You were already able to have a project that combined Arabic, Chinese and Spanish, but now any of the report options will show all these characters properly. Please note, this does require that your computer has font and language packs to support the scripts you want to work with, but most operating systems now include these by default.

 

We've also changed the way that Quirks are scaled, so now it's easier to see the different sizes with small and large projects.

Other minor improvements include:

  • A fix to DOCX exporting, so that this opens properly in more versions of Word
  • Better display of long property values
  • Better scrolling of the canvas area if zoomed in
  • Improvements to scrolling in Windows 8

 

If you already have Quirkos installed, you can update just by downloading the new version and installing over the old one (by default at c:\\Program Files (x86)\\quirkos-1.0 in Windows). Your files, activation and recently used file list will be unaffected. However, from now on, Quirkos will install in Windows in C:\\Program Files (x86)\\Quirkos so that new versions will automatically over-write the old one.

 

There is no change to the file format, so there are no compatibility issues between v1.0 and v1.1, and the update is free for existing customers.

 

Do let us know if you have any suggestions or feedback, many of these improvements were requested by users, so it is definitly worth getting in touch. We hope these new features help with your research, and look out for our Linux and Android versions, which should be publicly available at the end of the month.

Spring software update for Quirkos

snowdrops

Even in Edinburgh it’s finally beginning to get warmer, and we are planning the first update for Quirkos. This will be a minor release, but will add several features that users have been requesting.


The first of these is a batch import facility, you will be able to import a whole folder of text files, or just multiple files at once. This will be very useful for bringing in all your transcripts in one go, or for importing data from existing research projects.


Secondly, we are improving support for non-Latin based scripts, so that reports and other outputs will be able to show different languages such as Arabic, Chinese, Hebrew and Korean to name a few. Quirkos already allows you to work with all these languages in your project, but now generated reports will show all of these scripts properly.


We’ve also made a series of tweaks and alterations to the software, which will improve usability. These include fixing some small display issues with longer source properties, and scrolling on Windows 8 touchscreen devices. There should also be a few speed improvements, and better growth of quirks as text is added to them. This will make differences between emerging themes much more obvious.


Finally, we are really excited that this release will add two new platforms for Quirkos. First is our Android ‘app’, designed specifically for tablets. Unlike any other qualitative software for mobile platforms, the Android version of Quirkos has exactly the same feature set as the full desktop/laptop version. This means you can not only code on the go, on a great touchscreen interface, but also generate reports, run searches and add sources. Files are completely compatible across all platforms, so you can start work on your laptop, send your project to your tablet to do some coding on the train, and finish it off at home on your desktop.


Please note, that while there is technically nothing to stop you using this on your phone as well, it obviously becomes very fiddly on small screens! I’ve used Quirkos on a 4” phone, and while everything works, you’d need very small fingers and a lot of patience! Having said that, pair a Bluetooth mouse with a phone or tablet, and you quickly have a very flexible and portable coding tool.


Finally, I am thrilled to be able to release the first commercial qualitative software package for Linux! I’ve long been a user and advocate of Linux in all it’s different flavours, so supporting it is a great step to allow people to work on any platform they like. Again, the features and file compatibility will be identical, but with all the stability and security that Linux offers. While there are already two great open-source packages for Linux, the RQDA plugin for R is best for statistical analysis, and the lovely Weft QDA hasn’t been updated in nearly a decade. 


This update will be free for all existing users, and any new downloads will include the latest release. It doesn’t change the file structure at all, so there will be no compatibility problems. We hope to have this all ready and tested for you in March, so keep following this blog for the latest announcements!

 

How to organise notes and memos in Quirkos

EraserGirl post-it-notes

 

Many people have asked how they can integrate notes or memos into their project in Quirkos. At the moment, there isn’t a dedicated memo feature in the current version of Quirkos (v1.0), but this is planned for a free upgrade later in the year.


However, there are actually two ways in which users can integrate notes and memos into their project already using methods that give a great deal of flexibility.


The first, and most obvious ‘workaround’ is to create a separate source for notes and memos. First, create a blank source by pressing the (+) button on the bottom right of the screen, and select ‘New Source’. In the source properties view (Top right) you can change the name of this to ‘Memos’ or ‘Thoughts’ or something appropriate. You can then edit this source by long-clicking or right clicking in the source and selecting ‘Edit Source Text’. Now you have a dialogue box into which you can keep track of all your thoughts or memos during the coding process, and keep coming back to add more.


The advantage to having your memo as a source is that you can code with it in exactly the same way you would with any of your other sources. So you can write a note ‘I’m not sure about grouping Fear and Anxiety as separate codes’ and actually drag and drop that text onto the Anxiety and Fear bubbles – assigning that section of your note as being about those categories. When running queries or reports, you can easily see your comments together with the coding for that source, or just look at all your notes together.


This approach is most useful if you want to record your thoughts on the coding process or on developing your analysis framework. You can also have a series of note sources – for example if you had several people coding on a project. Don’t forget that you can export a source as a Word file with all the annotations, should you want to print or share just your notes on a project. One further tip is to create a Yes/No source property called ‘Memo’ or ‘Note’ so you can record which source(s) contain memos. Then when running queries or reports you can quickly choose whether to include coded memos or not.


However, if you want to record specific notes about each source, the second method is to actually create a source property for comments and notes. So for example, you might want to record some details of the interview that might have contextual importance. You can create a source property for ‘Interview conditions’ and note things like ‘Noisy room’ and ‘Respondent scared by Dictaphone’. By changing this property to be multiple choice, you can record several notes here, which of course can be used again across all the sources. This would let you quickly mark which interviewees were nervous about being recorded, and even see if responses from these people differed in a query comparison view.


However, you can also have a source category for more general notes, and add as many values to this property as you like. At the moment you can have very long values for source properties, but more than the first few words will not be shown. We are going to change this in an update in the next few weeks that will allow you to view much longer notes stored as property values.


These two different approaches should allow you plenty of ways to record notes, memos and musings as you go through and analyse your project. They also give you a lot of ways to sort and explore those notes – useful once you get to the stage of having lots of them! In future releases we will add a specific memo feature which will allow you to also have the option to add a note to a specific coding event, and will be implemented in unique but intuitive way. Watch this space!

The dangers of data mining for text

 Alexandre Dulaunoy CC - flickr.com/photos/adulau/12528646393

There is an interesting new article out, which looks at some of the commonly used algorithms in data mining, and finds that they are generally not very accurate, or even reproducible.

 

Specifically, the study by Lancichinetti et al. (2015) looks at automated topic classification using the commonly used latent Dirichlet allocation algorithm (LDA), a machine learning process which uses a probabilistic approach to categorise and filter large groups of text. Essentially this is a common approach used in data mining.

 

But the Lancichinetti et al. (2015) article finds that, even using a well structured source of data, such as Wikipedia, the results are – to put it mildly, disappointing. Around 20% of the time, the results did not come back the same, and when looking at a more complex group of scientific articles, reliability was as low as 55%.

 

As the authors point out, there has been little attempt to test the accuracy and validity of these data mining approaches, but they caution that users should be cautious about relying on inferences using these methods. They then go-on to describe a method that produces much better levels of reliability, yet until now, most analysis would have had this unknown level of inaccuracy: even if the test had been re-run with the same data, there is a good chance the results would have been different!

 

This underlines one of the perils with statistical attempts to mine large amounts of text data automatically: it's too easy to do without really knowing what you are doing. There is still no reliable alternative to having a trained researcher and their brain (or even an average person off the street) reading through text and telling you what it is about. The forums I engage with are full of people asking how they can do qualitative analysis automatically, and if there is some software that will do all their transcription for them – but the realistic answer is nothing like this currently exists.

 

Data mining can be a powerful tool, but it is essentially all based on statistical probabilities, churned out by a computer that doesn't know what it is supposed to be looking at. Data mining is usually a process akin to giving your text to a large number of fairly dumb monkeys on typewriters. Sure, they'll get through the data quickly, but odds are most of it won't be much use! Like monkeys, computers don't have that much intuition, and can't guess what you might be interested in, or what parts are more emotionally important than others.

 

The closest we have come so far is probably a system like IBM's Watson computer, a natural language processing machine which requires a supercomputer with 2,880 CPU cores, 16 terabytes of ram (16,384GB), and is essentially doing the same thing – a really really large number of dumb monkeys, and a process that picks the best looking stats from a lot of numbers. If loads of really smart researchers programme it for months, it can then win a TV show like Jeopardy. But if you wanted to win Family Feud, you'd have to programme it again.

 

Now, a statisical overview can be a good place to start, but researchers need to understand what is going on, look at the results intelligently, and work out what parts of the output don't make sense. And to do this well, you still need to be familiar with some of the source material, and have a good grip on the topics, themes and likely outcomes. Since a human can't read and remember thousands of documents, I still think that for most cases, in-depth reading of a few dozen good sources probably gives better outcomes than statistically scan-reading thousands.

 

Algorithms will improve, as outlined above, and as computers get more powerful and data gets more plentiful, statistical inferences will improve. But until then, most users are better off with a computer as a tool to aid their thought process, not to provide a single statistic answer to a complicated question.

 

Help us welcome Kristin to Quirkos!

So far, Quirkos users have mostly been based in the academic and university based research areas: perhaps not surprising considering where the project grew from. However, from very early on we got a lot of positive feedback from market research companies working with qualitative and text based data, who had many of the same frustrations and issues with qualitative research software that we had in the academic sphere. Indeed, some of the early alpha-testers of Quirkos were based in a typical small, independent market research firm.

 

But it's not really possible to lump all of these groups of researchers together, they have different needs; not just in terms of features in the software (although most of these are very similar), but also in terms of support and case studies. Qualitative market researchers need to engage with their clients in a different way, often using dynamic and visual approaches that Quirkos is ideally suited for.

 

So, to this end, we are very excited to announce a new recruit to the Quirkos offices: Kristin Schroeder, who will be focusing on market research and commercial users. Kristin studied Modern History at Merton College, Oxford, but is a native to the Baltic coast in Germany, and an avid sci-fi fan. She brings with her nearly a decade of sales experience working in Northern Ireland with large commercial clients for global automotive supplier Ryobi. Her extensive track record of international engagement will enable us to work better with users in the UK and abroad.

 

This will allow Daniel to continue his focus on supporting the researchers he knows best, in academia and the public sector, while Kristin can help Quirkos grow into new areas, helping more researchers across the globe to find answers to their questions.

 

New Leith offices for Quirkos

Just in time for the new year, Quirkos is growing!

 

We now need a bigger office to accomodate new hires, so we've moved to the 'Shore' at Leith, the seafront of Edinburgh. We've now got space to grow further, and to entertain visitors, all within walking distance of the sea and a short trip from the centre of Edinburgh. There are many exciting companies around us, and we are happy to be in such a nice part of town, with a different place for lunch and coffee every day of the month!

 

Our new address is:

27 Ocean Drive
Leith
Edinburgh
EH6 6JL

And we've got a new phone number too, 0131 555 3736

 

If you are coming to visit us, please let us know in advance, but the best bet is to set your sat-nav for Tower Place, a little cul-de-sac next to us which usually has some parking. We are just on the corner with Ocean Drive.

Happy new year to you all, and hope to see you soon!

Don't share reports with clients, share your data!

When it comes to presenting findings and insight with colleagues and clients, the procedure is usually the same. Create a written summary report, deliver the Powerpoint presentation, field any questions, repeat until everyone is happy.

 

But this approach tends to produce very static uninspiring reports, and presentations that lack interaction. This often necessitates further sessions, if clients or colleagues have questions that can't be directly answered, want additional clarifications, or the data explored in a different way. And the final reports don't always have the life we'd want for them, ending up on a shelf, or buried in a bulging inbox.

 

But what if rather than sharing a static report, you could actually share the whole research project with your clients? If rather than sending a Powerpoint deck, you could send them all of the data, and let them explore it for themselves? That way, if one of the clients is interested in looking at results from a particular demographic group, they can see it themselves, rather than asking for a report to be generated. If another client wants to see all the instances of negative words being used to describe their brand, they can see all the quotes in one click, and in another all the positive words.

 

In many situations, this would seem like an ideal way to engage with clients, but usually it cannot be facilitated. To send clients a copy of all the data in the project, transcripts, nodes, themes and all would be a huge burden for them to process. Researchers would also assume that few clients would be sufficiently versed in qualitative analysis software to be able to navigate the data themselves.

 

But Quirkos takes a different approach, which opens up new possibilities for sharing data with end users. As it is designed to be usable by complete novices at qualitative research, your project file, and the software interface itself can be used as a feedback tool. Send your clients the project data in a Quirkos file, with a copy of the software that runs live from a USB stick. You can even give them an Android tablet with the data on, which they can explore with a touch interface. They can then quickly filter the data however they like, see all the responses you've coded, or even rearrange your themes or nodes in ways that makes sense for them. The research team have collected the data, transcribed and coded it, but clients can get a real sense of the findings, running searches and queries to explore anything of interest to them.

 

And even when you are doing a presentation, while Quirkos will generate visual graphs and overviews of the data to include as static image files in Powerpoint, why not bring up Quirkos itself, and show the data as a live demonstration? You can show how themes are related, run queries for particular demographics segments, and start a really interactive discussion about the data, where you can field answers to queries in real time, generating easy to understand graphical displays on the fly. Finally, you can generate those static PDF or Word reports to share and cement your insights, but they will have come as a the result of the discussion and exploration of the project you did as collaborators.

 

Isn't it time you stopped sharing dry reports, and started sharing answers?