What’s in the Black-Box of the Quirkos Qualitative Researcher Journeys Project?

In funded research, the focus is on research outcomes and findings. Somebody has paid for the research and they want to know the answers. Fewer people want to know how it was done. Research processes are messy, and in qualitative research, they are usually very ‘wordy’, so if and when it is presented, it is selected and summarised. The process therefore is often hidden from view and difficult to get in to, so it is commonly described as a ‘black-box.’* When learning about qualitative research methodology, novice researchers want to see and use practical examples of how research was undertaken that go beyond the kind of overview found in research outputs; they want to see inside the research black-box.

These usually hidden parts of the process can include:

• How was ethical approval gained? What did the application for the IRB/ethics board look like?
• How were people recruited? What did the advertising, information and consent forms look like? How many people declined, and why?
• What kind of notes did the researcher make on the process, or reflections from interviews?
• What did the transcriptions look like? How much did they have to be corrected?
• What does qualitative analysis actually look like? How many codes/themes where there? How long did it take? How many different approaches were tried?
• How did the researcher decide which were the important things to write up?
• What journals were approached, how many revisions, and how many rejections were there?

All these things are edited out of the final outputs of research projects, but are essential parts of the research process for new researchers to learn. While textbooks and research training programmes may teach the theory of the process (you need to have a consent form) they rarely show an example. Most textbooks describe concepts like thematic analysis at great length - but never actually show you what it looked like for a project - all the codes and themes that were used or rejected.

Some black-box items of the research process will typically be seen in thesis appendices, or are presented at a conference. As Quirkos is an analysis tool, it is unsurprising that we here at Quirkos also devote a lot of our blogs to the process of the analysis of data. But that's not enough - it's easier to learn by example.

In keeping with Quirkos’s philosophy of accessibility, Quirkos’s Qualitative Researcher Journey project (QRJP) aimed to generate a pseudonymised open access data-set that can be used to teach and learn qualitative research methods. We uploaded transcripts from the project for open access to our data directory: https://www.quirkos.com/workshops/qual/. Now we are furthering our open access ethos by publishing the analysis files and project documentation there too.

Documents include:

·       Design and approval documents

·       Advertising and recruitment web pages and emails

·       Data gathering aide-memoire and pro-forma

·       Data analysis summaries

Data analysis files with first pass and later iterative coding are also provided, so that researchers can get a sense of how coding evolves during the analysis process.

Table showing parts of the research process of Qualitative Researcher Journeys Project. (QRJP).

One may argue that some of this is not black-box at all; the information was at some point in the public domain, especially things like advertising and calls for participants. However, it is rarely presented alongside project material for publication and is frequently discarded, or treated with less care once it has done its job; very much like the dust jacket of hard-back books. You may find a first edition book, but does it have its dust jacket? Probably not. I include this ‘research dust jacket’ material in the black-box, because that’s literally and figuratively where so much of it ends up once a project ends.

Increasingly, funding bodies are creating repositories (see our blog on Examples of Qualitative Data) and requiring project teams to deposit research data into a repository. This can be re-used for related research projects, and the processes of research are made more transparent. Quirkos is in the process of submitting Qualitative Researcher Journey data and analysis files to the UK Data Service Qualibank. But project documents are not the only items in a black-box. The decisions made and the justifications are rarely aired, even if they have been recorded in a systematic way.

As QRJP was designed for use in teaching, Cathy Gibbons (Quirkos' Qualitative Research Specialist) will be opening the QRJP black-box to scrutiny at the virtual The Qualitative Report (TQR) Conference Feb 2023. Discussions will include:

·       What project documentation needs to cover for ethical purposes

·       How project proforma can be used to support data gathering and analysis processes

·       Some of the problems and limitations of real project documentation

·       What might be considered as minimally essential for good practice, as compared with helpful, practical, maximally possible, or un-necessarily burdensome.

Registration is free for general attendees, so why not come along? https://www.nova.edu/academic-affairs/tqr-annual-conference/index.html

Resources

https://www.quirkos.com/workshops/

*The term black-box is probably most well known from  'black box' aircraft flight voice recorders (they are actually red). It is a common metaphor, generally meaning  "a system or process that uses information to produce a particular set of results, but that works in a way that is secret or difficult to understand".(Cambridge Dictionary).