Evaluating feedback

We all know the score: you attend a conference, business event, or training workshop, and at the end of the day you get a little form asking you to evaluate your experience. You can rate the speakers, venue, lunch and parking on a scale from one-to-five, and tick to say whether you would recommend the event to a friend or colleague.

But what about the other part of the evaluation: the open comments box? What was your favourite part of the day? What could we improve for next time? Any other comments? Hopefully someone is going to spend time typing up all these comments, and see if there are some common themes or good suggestions they can use to improve the event next year. Even if you are using a nifty on-line survey system like SurveyMonkey, does someone read and act on the suggestions you spent all that time writing?

And what about feedback on a product, or on service in a hotel or restaurant? Does something actually happen to all those comments, or as one conference attendee once suggested to me, do they all end up on the floor?

In fact, this is a common problem in research. Even when written up, reports often just stay on the shelf, and don't have influence on practice or procedure. If you want decision makers to pay attention to participant feedback and evaluations, then you need to present them in a clear and engaging way.

 

For the numerical or discrete part of surveys, this is not usually too hard. You can put these values in Excel, (or SPSS if you are statistically minded) and explore the data in pivot tables and bar graphs. Then you can see that the happiest attendees were the ones who ranked lunch as excellent, or that 76% of people would recommend the day to others.

Simple statistics and visualisations like this are a standard part of our language: we hear and see them in the news, at board meetings, even in football league tables. They communicate clearly and quickly.

But what about those written comments? In Excel you can't really see all the comments made by people who ranked the conference poorly, or see if the same suggestions are being made about workshop themes for next year.

That's what Quirkos aims to do: become the 'Excel of text'. It's software that everyone can use to explore, summarise and present text data in an intuitive way.

If you put all of your conference evaluations or customer feedback in Quirkos, you can quickly see all the comments made by people who didn't like your product. Or everything that women from the ages of 24-35 said about your service compared with men from 45-64. By combining the numerical, discrete and text data, you have the power to explore the relationships between themes and the differences between respondees. Then you can share these findings as graphs, bubble maps or just the quotes themselves: quick and easy to understand.

This unlocks the power of comments from all your customers, because Quirkos allows you to see why they liked a particular product. And it gives you the chance to be a better listener: if your consumers have an idea for improving your product, you can make it pop out as clear as day.

Hopefully it also breaks a vicious circle: people don't bother leaving comments as they assume they are aren't being read, and thus organisers stop asking for comments, because those sections are ignored or give generic responses.

 

So hopefully next time you fill out a customer feedback form or event evaluation, your comments will lead to direct improvements, rather than just being lost in translation.

Touching Text

Presenting Quirkos at the CAQDAS 2014 conference this month was the first major public demonstration of Quirkos, and what we are trying to do. It’s fair to say it made quite a splash! But getting to this stage has been part of a long process from an idea that came about many years ago.

Like many geeks on the internet, I’d been amazed by the work done by Jeff Han and colleagues at the University of New York on cheap, multi-touch interfaces. This was 2006, and the video went viral in a time before iPhones and tablets, when it looked like someone had finally worked out how to make the futuristic computer interface from Minority Report which had come out in 2002. Others, such as Johnny Lee at Carnegie Mellon University had worked out how the incredible technology in the controllers for the Wii could make touchscreen interactive whiteboards with a £25 toy.

I’ve always been of the opinion that technology is only interesting when it is cheap: it can’t have an impact when it’s out of reach for a majority of people. Now, none of this stuff was particularly ground-breaking in itself, but these people were like heroes to me, for making something amazing out of bits and pieces that everyone could afford.

Meanwhile, I was trying to do qualitative analysis for my PhD [danfreak.net/thesis.html], and having a iBook that wouldn’t run any of the qualitative analysis packages, I cobbled together my own system: my first attempt at making a better qualitative research system. It was based on a series of unique three letter unique codes I’d insert into a sentence, and a Linux based file search system called ‘Beagle’ which allowed me to see a piece of text I’d assigned with a code across any of the files on my computer. Thus in one search I could see all the relevant bits of text from interviews, focus groups, diaries and notes. It was clunky, but worked, and was the beginning of something with potential.

 

By 2009, I had my first proper research job in Oxford, and was spending my salary trying to make a touchscreen computer out of a £120 netbook and a touchscreen overlay I’d imported from China. In fact, I got through two of these laptops, after short-circuiting the motherboard of one while trying to cram the innards into a thin metal case. What excited me was the potential for a £150 touchscreen computer, with no keyboard, that you used like a ‘tablet’ from Star Trek. Then, while I was doing this, Apple came out with the long-anticipated iPad, which had the distinct advantage of being about ¼ of the thickness and weight!

But while all this was going on in my spare time, at work I was spending all day coding semi-structured interviews for a research project. I was being driven mad with the slow coding process, Nvivo was crashing frequently and corrupting all the work when it did, and using interfaces in the 21st century that were beginning to feel a whole generation behind.

And that’s where the idea came from: me speculating on what qualitative analysis would be like with a touch screen interface. What if you could do it on a giant tablet or digital whiteboard with a team of people? I drew sketches of bubbles (I’ve always liked playing with bubbles) that grew when you added text to them, integrating the interface and the visualisation, and showing relationships between the themes.

 

After this, the idea didn’t really progress until I was working on my next job, at Sheffield Hallam University. Again, qualitative analysis was giving me a headache, this time because we wanted to do analysis with participants and co-researchers, and most of the packages were too difficult to learn and too expensive to afford to let the whole team get involved. A new set of colleagues shared my pain with using current CAQDAS software, and as no-one else seemed to be doing anything about it, I thought it was worth giving a try.

I took a course in programming user interfaces using cross-platform frameworks, and was able to knock up some barely functioning prototypes, at the time called ‘Qualia’. But again, things didn’t really progress until I left my job to focus on it full time, fleshing out the details and hiring the wonderful Adrian Lubik: a programmer who actually knows what he’s doing!

With the project gaining momentum, a better name was needed. Looking around classical Greek and Latin names, I came across ‘kirkos’, the Greek word which is the root of the word ‘circle’. Change the beginning to ‘Qu’ for qualitative, and voilá, Quirkos was born: Qualitative Circles. Something that very neatly summed up what I’d been working towards for nearly a decade.

In June we’ll be releasing the beta version to testers for the first time, and the final version will go on sale in September at a lower price point that means a lot more people can try qualitative research. It’s really exciting to be at this stage, with so much enthusiasm and anticipation building in the market. But it’s also just a beginning; we have a 5 year plan to keep adding unique features and develop Quirkos into something that is innovative at every stage of the research process. It’s been a long journey, but it’s great that so many people are now coming along!