Using Quirkos for Systematic Reviews and Evidence Synthesis

Most of the examples the blog has covered so far have been about using Quirkos for research, especially with interview and participant text sources. However, Quirkos can take any text source you can open on your computer, including text PDFs (but not scanned PDFs where each page is effectively a photograph). So why not use Quirkos like a reference manager, to sort and analyse a large cohort of articles and research? The advantage being that you can not only keep track of references, but also cross-reference the content: analysing common themes across all the articles.

There are two ways to manage this: first you can set the standard information for each source/article that is imported, such as author, year, journal, etc. If you format these as you wish them to appear in the reference (by putting the commas and dots in the value), and order them with the Properties and Value editor, you can create reports that churn out the references in whichever notation you need, such as Harvard or APA. But you can also add any extra values you like at an article level, so you could rank articles out of 10, have a comment property, or categorise them by methodology. This way, you can quickly see only text from articles rated as 8/10 or above, or everything with a sample size between 50 and 100: whatever information you categorise.

Secondly you can categorise text in the article using Quirk bubbles. So as you read through the articles, code sections in any way that is of interest to you: highlight sections on the methodology, bits you aren’t convinced about, or other references you want to check out. Highlight findings and conclusion sections (or just interesting parts of them), and with the properties you can quickly look at all the findings from papers using a particular approach, and compare and contrast them. It’s obviously quite a bit of work to code all your articles, but since you would have to read through all the papers anyway, making your notes digital and searchable in this way makes it much quicker and flexible when pulling it all together.

With qualitative synthesis you can combine multiple pieces of research, and see if there are common themes, or contradictions. Say you have found three articles on parenting, but they are all from different minority ethnic communities. Code them in Quirkos, and in a click you can see all the problems people are having with schools across all groups, or if one community describes more serious issues than another.

Evidence synthesis and systematic reviews like this are often, and quite rightly, mandated by funders and departments before commissioning a new piece of research, to make sure that the research questions add meaningfully to the existing canon. However, it’s also worth noting that, especially with qualitative synthesis taken from published articles, there can be a publication bias by relying only on comments left in the final paper: most of the data set is hidden to secondary researchers. Imagine if you are looking at schooling and parenting, but are taking data from an article on the difficulties of parenting: it’s possible that the researchers did not include quotations on the good aspects of school as it was outside the article’s focus. If possible it’s always worth getting the full data set, but this can often throw up data protection and ethical issues. There’s no simple answer to these problems, except to make sure readers are aware of your sources, and anticipate the likely limitations of your approach. Often with qualitative research, it feels like reflexivity and disclaimers go hand in hand!

Getting a foot in the door with qualitative research

A foot in a doorA quick look at the British Library thesis catalogue suggests that around 800 theses are completed every year in the UK using qualitative methods*. This suggests that 7% of the roughly 10,000 annual British PhDs completed use qualitative methods. There are likely to be many more masters theses, and even undergraduate dissertations that use some qualitative methods, so the student demand for qualitative training is considerable.


Usually, while PhD Research Training Programmes will include good coverage of different qualitative methods and ethical issues, using software for qualitative analysis is often not covered. In my experience it is either left to summer school sessions, annual one-off internal training sessions, but usually training at an external full or two day session at organisations like the University of Surrey CAQDAS programme. Most PhD students (especially in the UK) are at considerable time and financial pressures, so accessing this training is often difficult. Again, it's sometimes difficult to get a foot in the door with qualitative analysis software.


Yet there are some good opportunities for qualitative researchers, even outside academia. Obviously market research is a huge employer, and can provide very varied work, changing with every project. Increasingly it seems that the public sector, at both the local and national level are hiring researchers with qualitative experience, especially in organisations like the NHS, where patient satisfaction is becoming an increasingly important metric.


Quirkos has been designed with my own experiences in mind, to provide an easy way to get started with qualitative analysis. In fact, I've jokingly referred to it before as a 'gateway' product, easy to start, and hopefully leading to a good experience and a desire to progress to advanced ways of working! We are also going to offer PhD students a discounted 'PhD Pack', which will include a student licence, on-line training, and two academic licences for their supervisors, so that the whole team can see the progress and comment on ongoing analysis.


Researching the numbers of students in the UK, I was stunned to find out that the number of full-time PhD students has nearly doubled, from 9,990 in 1997 to 18,075 in 2010 – the last year for which statistics are available. Now, clearly the number of academic positions has not increased at the same rate, (although it has increased over that time period) so the number of available academic jobs has not kept up with supply. Of course, a PhD can lead to many more opportunities, but it is clear that there is great competition for post-doctorate posts. This has been noted by many other commentators, but also in my own experience. Many of my post-doc friends and colleagues are ridiculously intelligent and capable people, but are still in jobs that chronically undervalue their abilities. Between ourselves, we often joke that for academic jobs, it has become a game of 'dead-man's-boots', waiting for a senior academic to retire, starting a chain of departmental promotions that create a new junior position. These posts are also only available after doing several temporary post-doc positions: it is a long process to get your foot in the door, and you often find yourself competing with good friends.


It seems to me that many university departments are now scaling back the number of PhD and Masters students they accept, acknowledging the pressure that large student numbers put on supervisors, despite the large amounts of income they bring to the department (especially Masters programmes). However, if widespread, this change is not yet visible in the latest HEFCE data, which dates back to 2010-11, and shows higher numbers of starters, and an increase in (projected) completion rates. Yet there is a huge and growing pool of very bright critical thinkers on the market, and even if academic opportunities are limited, a good number of other doors to get a foot-hold into.


* To get these figures, I have only used search terms qualitative AND either interview or “focus group” across titles and abstracts, to make sure that no other uses of the phrase were included: for example genetic qualitative research. Other methods such as ethnography and diaries added only a dozen or so results each. Frustratingly, the EThOS search doesn't let you specify a date range, but including a year (2012) as a search term mostly returns submissions from that year. It's also interesting to note that the number of PhDs mentioning qualitative methods has doubled since 2007, although it's difficult to tell how much of this is due to any increased popularity of qualitative research, or the increase in total submissions noted above, and the increase in digital submissions to the BL system.

Share on Facebook Share on Twitter