Delivering qualitative market insights with Quirkos

delivering fashion

 

To build a well-designed, well-thought-out, and ultimately useful product, today’s technology companies must gain a deep understanding of the working mentality of people who will use that product. For Melody Truckload, a Los Angeles tech company focused on app-based freight logistics, this means intense market research and a focus on training sales agents as researchers.

 

Kody Kinzie, director of Melody’s special research and operations team, Cythlin Intelligence, was faced with introducing qualitative social research and analysis to people who had never considered themselves researchers before.

 

“Quirkos was the first truly accessible qualitative program I found,” Kinzie said.

 

Quirkos was designed with the philosophy that anyone can become a qualitative researcher. The goal is to allow companies and agencies to adopt unique ways to understand their staff and the wider marketplace. By making qualitative data visual and easy to code, users can see their results emerge and gain quick overviews of complex issues.

 

Companies like Melody are at the forefront of developing the next generation of qualitative insight, and Quirkos is helping to open the door to innovative new methods of business intelligence.

 

Kinzie started training his team members to use Quirkos but said he soon discovered that the simple coding allowed even a novice to develop complex data structures with notable uniqueness. Often, he found that these code structures were well suited to analyzing particular elements the researchers were interested in, and he began documenting the experiment to evaluate the resulting structures.

 

Here’s how Kinzie and his team use Quirkos:
One team member will send a Quirkos database to another team member — a researcher who examines the code structure and walks the requesting team member through an explanation of the thought process that went into creating the code. The data structure’s strengths and weaknesses are then assessed and distilled into a report. The researcher examines Melody’s code construction to discover what kind of information it is most effective at analyzing or categorizing, as well as whether the code tags and organizes information or clusters information into meaningful relationships.

 

This helps researchers understand what kind of questions these information structures should be applied to, and where a particular researcher’s methods might excel. The ability to use Quirkos to build and analyze unique and flexible databases from these structures has given Melody an edge in developing and sharing insights throughout the team.

 

While Melody Truckload’s app currently wraps up beta testing with commercial partners, the Quirkos approach has been put to the test most recently on the Melody team’s latest project, Melody Fashion.

 

“In the complex world of L.A.’s Fashion District, which is the part of town that houses the city’s fashion industry wholesale market, freight consolidation desperately needs to be modernized,” Kinzie said. “The objective of Melody Fashion is to provide a platform for fulfilment and consolidation that takes into account a detailed understanding of a market with many players.”

 

To that end, sales agents were trained to analyze interactions using grounded theory on Quirkos and to aggregate data garnered in their interactions with customers. It led to valuable insights, including a partnership with local shipping experts to bring Melody Fashion’s technology to the district.

 

Melody operations manager Marcus Galamay, who introduced new agents to Quirkos software and guided them through their first qualitative exercises, said, “Quirkos provides an intuitive introduction to qualitative analysis for our sales agents, augmenting their role in a way that’s expanded our insights into our client base. It’s a niche that many might not think to pursue, but it’s already delivered results in terms of better understanding of the data we generate and refining our market strategy based on that.”

 

Thanks to its ease of use and its powerful ability to assist in important social research, Quirkos was instrumental in providing Melody with the insight necessary to build smart and useful technology for a distinct and totally new customer base.

 

 

Bing Pulse and data collection for market research

bing pulse example

 

Judging by the buzz and article sharing going on last week, there was a lot of interest and worry about Microsoft launching their own market research platform. Branded as part of ‘Bing’, their offering, called ‘Pulse’ has actually been around for a while, and is still geared around collecting feedback from live events, especially political discussions.


I can see why this move might have a lot of companies worried, it seems to me that the market research arena is crowded with start-ups and established firms offering platforms, or ‘communities’ for collecting participant data. There’s LiveMinds, Aha!, VisionLive, a quick search will bring up dozens of competitors. So an entry into the market from an organisation with deep pockets and brand awareness like Microsoft may well have many looking to see how this develops. However, with my own limited time with Pulse, I don’t think there is much to worry about yet.


First of all, Pulse is currently entirely focused on one niche, feedback on live events. There are no tools to do anything like advert or creatives validation, no proper survey tools or interactive online focus groups. The MO is very much quantitatively focused, with very little option to capture qualitative feedback at this time. Secondly, it seems to have a lot of limitations, and in this beta state, almost no documentation.


I quickly got stuck trying to create a real-time voting question, with a mandatory box for ‘response theme’ that was greyed out, but wouldn’t continue without being completed. The ‘help’ tools just link to a generic Bing help website, which don’t contain any content about Pulse. The layout is a little confusing, getting you stuck in a strange loop between the ‘Live Dashboard’ and ‘Pulse Options’, and it’s also slow: get used to seeing the little flapping loading logo after every action.

 

As for integration, the only option at the moment seems to be the API, which only has four available calls. There doesn’t seem to be any way to get results (especially those not covered by those API calls) out from the platform: I can’t see any CSV export or the like. Also, considering the powerful analytic options available through the Azure platform, it’s disappointing not to see any easy integration there. In short, far from being a quick DIY solution, you will need someone to programme yet another API into your platform to do anything more than look at a few graphs on the Pulse platform.

 

I want to stress that this was hardly a detailed review and test of the capabilities of the platform, my opinions are based just on playing with it for an hour or so. However, it is nice to be able to try it out with just a registration, personally I don’t like products where the demo is locked away and difficult to try out. It’s a competitive market, and I feel more inclined to trust software that the developers aren’t shy of showing off!

 

Now, I understand that most market research providers are not so much worried about the current feature set of Pulse, but what this entry into the field means in the future, especially for a product that Microsoft is content to offer for free at this time. But I would echo some of the comments made in the Greenbook article by Leonard Murphy, that it usually doesn’t make sense for market research firms to do their own their own quantitative data collection. The future, he says, is integrating with data collection tools and adding value in terms of insight, custom development and consultation.


And that is the crux with all these market research platforms: they are primarily data collection tools, with limited analytics. Pulse doesn’t seem to have anything on this front at the moment, but with too many of these solutions, the insight stops with a couple of graphs or statistics. I feel there is still the need to integrate with another tool, or draw from extensive market research analytic experience to make anything from the data once it has been collected. It maybe that most clients don’t expect or require any kind of rigour in the breakdown of project results, especially when it comes to qualitative data. I am still yet to see anything that looks to me like a true end-to-end platform for market research, but am willing to be proved wrong!

 

At the moment, there are some great and flexible tools for collecting customer data online, be it quantitative or qualitative. But these are ubiquitous, and very cheap to run – we host an online survey platform for our customers for free, just as a convenience. Yet getting to answers and insight from that data usually requires an additional analytical step, especially for qualitative research. As I’ve said before,  the most difficult step is understanding the data and how you integrate analytics into your workflow. Increasingly the data collection platform you choose, and how much you pay for it will not be an issue.

 

 

Help us welcome Kristin to Quirkos!

So far, Quirkos users have mostly been based in the academic and university based research areas: perhaps not surprising considering where the project grew from. However, from very early on we got a lot of positive feedback from market research companies working with qualitative and text based data, who had many of the same frustrations and issues with qualitative research software that we had in the academic sphere. Indeed, some of the early alpha-testers of Quirkos were based in a typical small, independent market research firm.

 

But it's not really possible to lump all of these groups of researchers together, they have different needs; not just in terms of features in the software (although most of these are very similar), but also in terms of support and case studies. Qualitative market researchers need to engage with their clients in a different way, often using dynamic and visual approaches that Quirkos is ideally suited for.

 

So, to this end, we are very excited to announce a new recruit to the Quirkos offices: Kristin Schroeder, who will be focusing on market research and commercial users. Kristin studied Modern History at Merton College, Oxford, but is a native to the Baltic coast in Germany, and an avid sci-fi fan. She brings with her nearly a decade of sales experience working in Northern Ireland with large commercial clients for global automotive supplier Ryobi. Her extensive track record of international engagement will enable us to work better with users in the UK and abroad.

 

This will allow Daniel to continue his focus on supporting the researchers he knows best, in academia and the public sector, while Kristin can help Quirkos grow into new areas, helping more researchers across the globe to find answers to their questions.

 

Don't share reports with clients, share your data!

When it comes to presenting findings and insight with colleagues and clients, the procedure is usually the same. Create a written summary report, deliver the Powerpoint presentation, field any questions, repeat until everyone is happy.

 

But this approach tends to produce very static uninspiring reports, and presentations that lack interaction. This often necessitates further sessions, if clients or colleagues have questions that can't be directly answered, want additional clarifications, or the data explored in a different way. And the final reports don't always have the life we'd want for them, ending up on a shelf, or buried in a bulging inbox.

 

But what if rather than sharing a static report, you could actually share the whole research project with your clients? If rather than sending a Powerpoint deck, you could send them all of the data, and let them explore it for themselves? That way, if one of the clients is interested in looking at results from a particular demographic group, they can see it themselves, rather than asking for a report to be generated. If another client wants to see all the instances of negative words being used to describe their brand, they can see all the quotes in one click, and in another all the positive words.

 

In many situations, this would seem like an ideal way to engage with clients, but usually it cannot be facilitated. To send clients a copy of all the data in the project, transcripts, nodes, themes and all would be a huge burden for them to process. Researchers would also assume that few clients would be sufficiently versed in qualitative analysis software to be able to navigate the data themselves.

 

But Quirkos takes a different approach, which opens up new possibilities for sharing data with end users. As it is designed to be usable by complete novices at qualitative research, your project file, and the software interface itself can be used as a feedback tool. Send your clients the project data in a Quirkos file, with a copy of the software that runs live from a USB stick. You can even give them an Android tablet with the data on, which they can explore with a touch interface. They can then quickly filter the data however they like, see all the responses you've coded, or even rearrange your themes or nodes in ways that makes sense for them. The research team have collected the data, transcribed and coded it, but clients can get a real sense of the findings, running searches and queries to explore anything of interest to them.

 

And even when you are doing a presentation, while Quirkos will generate visual graphs and overviews of the data to include as static image files in Powerpoint, why not bring up Quirkos itself, and show the data as a live demonstration? You can show how themes are related, run queries for particular demographics segments, and start a really interactive discussion about the data, where you can field answers to queries in real time, generating easy to understand graphical displays on the fly. Finally, you can generate those static PDF or Word reports to share and cement your insights, but they will have come as a the result of the discussion and exploration of the project you did as collaborators.

 

Isn't it time you stopped sharing dry reports, and started sharing answers?

 

Knowing your customers

barcode

As consumers, it feels like we are bombarded more than ever with opportunities for providing feedback on products and services. While shopping on-line, or even when browsing BBC News we are asked to complete a short questionnaire; after dealing with a telephone bank there’s the option to complete a quick survey; and at airport security you can rate your experience by hitting a button with either a smiley face or a frowny face.

 

But despite being told that ‘your feedback really matters to us’, what happens to it? It’s often difficult to see any change from your feedback, and even when giving direct feedback, too often the changes suggested are not made. But more than this, it’s sometimes difficult to understand how we can expect people to understand our needs and problems with such brief and forced categorisation. If I rate a telephone sales agent from 1-10 on categories of helpfulness, friendliness, and professionalism, you are forced to somehow shoehorn your feedback on any other aspect of the experience into these areas. You don’t know if ‘length of wait’ will be a category, and you could care less how friendly they were, if they couldn’t fix your problem.

 

I wonder this when hearing stories in the news about the continuing sales decline at Tesco, a huge organisation that clearly spends millions on customer understanding. Because for all the hoop-la around loyalty schemes, ‘convenience’ stores, price-matching (only with certain competitors), are management at some level blind to the rise of budget supermarkets? Customers clearly aren’t, and it’s difficult to tell if big supermarkets either have their head in the sand, or they assume their customers are stupid.

 

I can’t pretend to know why the likes of Lidl and Aldi have become so popular, it could just be price, but perhaps a more pleasant streamlined shopping experience, without having to choose between value, regular and luxury versions of everything. A focus group could tell you: if you had a wide range of users, you could ask them questions; or better still, let people raise issues themselves without being pigeonholed. And this seems to be more and more difficult for the grocery shopping market, thanks to a huge demographic shift. Watch Robert Preston’s excellent series on consumer culture, and you can see that in the 60s, to know retail shoppers was to know housewives: if you could get their spending, you got it all. But today everyone shops, and with a lingering recession, job pressures and a mobile work market, we shop whenever we get a chance. A one stop grocery shop usually tries to attract housewives, househusbands, students, bachelors, hipsters and skinflints alike, either spreading themselves too thin, or becoming bewildering to all.

 

So perhaps the one-size-fits all model is not going to be the way of the future. On-line grocery sales have begun to slow at just 5.1% of the market, either there is going to be an innovation here, or the market already has found its niche. But it seems that many smaller grocery stores are on the rise, tailored to a specific target audience. Health food shops which in the 90s used to only sell vitamins and gluten-free flour now sell 'healthy' cornflakes and fresh organic produce – so you can get everything in one shop. Marks and Spencer’s ubiquitous stores now cater for a new ready-meal elite: grabbing dinner for one or two on the way back from the office. And Farmfoods and Iceland have the low end of the market – frozen food so busy families can buy a week’s worth of inexpensive, easily prepared budget meals.

 

So these big chains do well by knowing their audience. And it’s no different for smaller, independent businesses. Quirkos is proposing that even these small firms can afford to do their own direct market research, and with detail that will give a much better feel for their customers than just relying on crude statistics and smiley and frowny faces. That way, rather than relying on very traditional market research, business can take a more local and individual approach. Rather than focus groups behind one-way-mirrors, or questionnaires with low engagement rates, why not invite a group of customers to a wine evening, and record a discussion about new products? If recorded properly by diligent staff, collated and analysed, informal feedback to cashiers can start to build a picture of what products or experiences are missing.

 

Then Quirkos would step in, providing software that is easy to get started with, so a manager can pull together all these sources of feedback, read them, and put them into themes that s/he can use to make the changes customers are looking for. Market research is already a huge industry in the UK, but can’t we go further and democratise it? Small scale for small businesses, quick to learn, and priced for everyone?

Getting a foot in the door with qualitative research

A foot in a doorA quick look at the British Library thesis catalogue suggests that around 800 theses are completed every year in the UK using qualitative methods*. This suggests that 7% of the roughly 10,000 annual British PhDs completed use qualitative methods. There are likely to be many more masters theses, and even undergraduate dissertations that use some qualitative methods, so the student demand for qualitative training is considerable.

 

Usually, while PhD Research Training Programmes will include good coverage of different qualitative methods and ethical issues, using software for qualitative analysis is often not covered. In my experience it is either left to summer school sessions, annual one-off internal training sessions, but usually training at an external full or two day session at organisations like the University of Surrey CAQDAS programme. Most PhD students (especially in the UK) are at considerable time and financial pressures, so accessing this training is often difficult. Again, it's sometimes difficult to get a foot in the door with qualitative analysis software.

 

Yet there are some good opportunities for qualitative researchers, even outside academia. Obviously market research is a huge employer, and can provide very varied work, changing with every project. Increasingly it seems that the public sector, at both the local and national level are hiring researchers with qualitative experience, especially in organisations like the NHS, where patient satisfaction is becoming an increasingly important metric.

 

Quirkos has been designed with my own experiences in mind, to provide an easy way to get started with qualitative analysis. In fact, I've jokingly referred to it before as a 'gateway' product, easy to start, and hopefully leading to a good experience and a desire to progress to advanced ways of working! We are also going to offer PhD students a discounted 'PhD Pack', which will include a student licence, on-line training, and two academic licences for their supervisors, so that the whole team can see the progress and comment on ongoing analysis.

 

Researching the numbers of students in the UK, I was stunned to find out that the number of full-time PhD students has nearly doubled, from 9,990 in 1997 to 18,075 in 2010 – the last year for which statistics are available. Now, clearly the number of academic positions has not increased at the same rate, (although it has increased over that time period) so the number of available academic jobs has not kept up with supply. Of course, a PhD can lead to many more opportunities, but it is clear that there is great competition for post-doctorate posts. This has been noted by many other commentators, but also in my own experience. Many of my post-doc friends and colleagues are ridiculously intelligent and capable people, but are still in jobs that chronically undervalue their abilities. Between ourselves, we often joke that for academic jobs, it has become a game of 'dead-man's-boots', waiting for a senior academic to retire, starting a chain of departmental promotions that create a new junior position. These posts are also only available after doing several temporary post-doc positions: it is a long process to get your foot in the door, and you often find yourself competing with good friends.

 

It seems to me that many university departments are now scaling back the number of PhD and Masters students they accept, acknowledging the pressure that large student numbers put on supervisors, despite the large amounts of income they bring to the department (especially Masters programmes). However, if widespread, this change is not yet visible in the latest HEFCE data, which dates back to 2010-11, and shows higher numbers of starters, and an increase in (projected) completion rates. Yet there is a huge and growing pool of very bright critical thinkers on the market, and even if academic opportunities are limited, a good number of other doors to get a foot-hold into.

 

* To get these figures, I have only used search terms qualitative AND either interview or “focus group” across titles and abstracts, to make sure that no other uses of the phrase were included: for example genetic qualitative research. Other methods such as ethnography and diaries added only a dozen or so results each. Frustratingly, the EThOS search doesn't let you specify a date range, but including a year (2012) as a search term mostly returns submissions from that year. It's also interesting to note that the number of PhDs mentioning qualitative methods has doubled since 2007, although it's difficult to tell how much of this is due to any increased popularity of qualitative research, or the increase in total submissions noted above, and the increase in digital submissions to the BL system.

Paper vs. computer assisted qualitative analysis

I recently read a great paper by Rettie et al. (2008) which, although based on a small sample size, found that only 9% of UK market research organisations doing qualitative research were using software to help with qualitative analysis.

 

At first this sounds very low, but it holds true with my own limited experiences with market research firms, and also with academic researchers. The first formal training courses I attended on Qualitative Analysis were conducted by the excellent Health Experiences Research Group at Oxford University, a team I would actually work with later in my career. As an aforementioned computer geek, it was surprising for me to hear Professor Sue Ziebland convincingly argue for a method they defined as the One Sheet of Paper technique, immortalised as OSOP. This is essentially a way to develop a grounded theory or analytical approach by reducing the key themes to a diagram that can be physically summarised on a single piece of paper, a method that is still widely cited to this day.

 

However, the day also contained a series of what felt like ‘confessions’ about how much of people’s Qualitative analysis was paper based: printing out whole transcripts of interviews, highlighting sections, physically cutting and gluing text into flipcharts, and dozens and dozens of multi-coloured Post-it notes! Personally, I think this is a fine method of analysis, as it keeps researchers close to the data and, assuming you have a large enough workspace, it lets you keep dozens of interviews and themes to hand. It’s also very good for team work, as the physicality gets everyone involved in reviewing codes and extracts.

 

In the last project I worked on, looking at evidence use for health decision making we did most of the analysis in Excel, which was actually easier for the whole team to work with than any of the dedicated qualitative analysis software packages. However, we still relied heavily on paper: printing out the interviews and Excel spreadsheets, and using flip-chart paper, post-its and marker pens in group analysis sessions. Believe me, I felt a pang of guilt for all the paper we used in each of these sessions, rainforests be damned! But it kept us inspired, engaged, close to the data and let us work together.

 

So I can quite understand why so many academics and market research organisations choose not to use software packages: at the moment they don’t have the visual connection to the data that paper annotations allow, it’s often difficult to see the different stages of the coding process, and it’s hard to produce reports and outputs that communicate properly.

 

The problem with this approach is the literal paper-trail – how you turn all these iterations of coding schemes and analysis sessions into something you can write up to share with others in order to justify how you made the decisions that led to your conclusions. So I had to file all these flip-charts and annotated sheets, often taking photos of them so they could be shared with colleagues at other universities. It was a slow and time consuming process, but it kept us close to the data.

 

When designing Quirkos, I have tried in some ways to replicate the paper-based analysis process. There’s a touch interface, reports that show all the highlighting in a Word document, and views that keep you close to the data. But I also want to combine this with all the advantages you get from a software package, not least the ability to search, shuffle dozens of documents, have more colours than a whole rainbow of Post-it notes, and the invaluable Undo button!

 

Software can also help keep track of many more topics and sources than most people (especially myself) can remember, and if there are a lot of different themes you want to explore from the data, software is really good at keeping them all in one place and making them easy to find. Working as part of a team, especially if some researchers work remotely or in a different organisation can be much easier with software. E-mailing a file is much easier than sending a huge folder of annotated paper, and combining and comparing analysis can be done at any stage of the project.

 

Qualitative analysis software also lets you take different slices through the data, so you can compare responses grouped by any caracteristics for the sources you have. So it's easy to look at all the comments from people in one location, or between a certain age range. Certainly this is possible to do with qualitative data on paper as well, but the software can remove the need of a lot of paper shuffling, especially when you have a large number of respondents.

 

But most importantly, I think software can allow more experimentation - you can try different themes, easily combine or break them apart, or even start from scratch again, knowing that the old analysis approach you tried is just a few clicks away. I think that the magic undo button also gives researchers more confidence in trying something out, and makes it easier for people to change their mind.

 

Many people I’ve spoken to have asked what the ‘competition’ for Quirkos is like, meaning, what do the other software packages do. But for me the real competitor is the tangible approach and the challenge is to try and have something that is the best of both worlds: a tool that not only apes the paper realm in a virtual space, but acknowledges the need to print out and connect with physical workflows. I often want to review a coded project on paper, printing off and reading in the cafe, and Quirkos makes sure that all your coding can be visually displayed and shared in this way.

 

Everyone has a workflow for qualitative analysis that works for them, their team, and the needs of their project. I think the key is flexibility, and to think about a set of tools that can include paper and software solutions, rather than one approach that is a jack of all trades, and master of none.