I recently wrote a chapter for the 2nd edition of The Practice of Qualitative Data Analysis
Research Examples Using MAXQDA that details a qualitative evaluation I did for the Coaching through Covid and Beyond programme using MAXQDA. The way the analysis was performed, how progress was documented, and how it is discussed and presented in the chapter, was informed by the Five-Level QDA method that Nick Woolf and I developed (2018). There wasn’t enough space in the chapter to include the illustrations and tables I used during the project, so this post provides those, as extra context to the chapter.
What is the Five-Level QDA Method?
The Five-Level QDA method is a CAQDAS pedagogy that Nick Woolf and I developed which highlights the importance of distinguishing between analytic strategies (what you plan to do) and software tactics (how you to do it). Our observations over many years of facilitating the learning and adoptions of qualitative software like MAXQDA showed that those who struggle to harness these tools powerfully often conflate strategies and tactics, with the common result being moving away from the software right at the moment when it could be most useful.
The Five-Level QDA method proposes that to move seamlessly between strategies (intentions around methodology and method) and tactics (appropriate use of CAQDAS-tools to operationalize strategies), it’s necessary to translate analytic tasks that sit at the strategies level into actions within the software (tactics) (Silver and Woolf, 2019). Doing this requires planning, and we’ve written about how this happens in several places – see references list at the end of this post.
Coaching through Covid analysis context
In March 2020 a call to action came for Coaches to deliver pro-bono (free) coaching to those working on the UK National Health Service (NHS) frontline during the crisis. Shortly afterwards the Coaching through Covid (CtC) Programme (later renamed Coaching through Covid and Beyond – CtCaB) was set-up, entirely run by volunteers. The programme offered free coaching sessions for NHS staff, to help those working at the frontline of the pandemic with the challenges they were facing. I joined the Core Strategy Team in December 2021, to provide research and evaluation support on a voluntary basis. The team wanted to understand whether and how the programme had been effective, from the perspective of both the NHS staff who received the coaching and the professional coaches who provided it. This was achieved by collecting feedback via two questionnaires; one administered to NHS staff, the other to Coaches. The chapter in the MAXQDA textbook discusses the analysis of the NHS staff feedback. For more information about the CtCaB programme see https://coachingthroughcovid.org
Planning an evaluation in real-time
The analysis began whilst the Programme was on-going, so the objectives were identified and adjusted as it progressed, in response to needs identified by the Core Strategy Team at points during the process. For example, at the outset the primary aim was to identify areas where provision could be improved, by focusing on aspects that Clients reported being less satisfied with.
Later on the focus shifted to consider lessons that could be learned from the whole process; i.e. from the Coaches and the Core Strategy team as well as the Clients, in order to share with others who may embark on a similar initiative in the future. These shifting objectives affected the way the analysis was undertaken.
Visualising the analysis at a high-level
The five Stages of analysis are illustrated in Figure 1. The chapter I wrote for the MAXQDA textbook discusses the first two stages only, so those are what I focus on in the rest of this post as well. However, as you can see from Figure 1. the analysis of the client feedback led to analysis of coach feedback and the two were integrated in the fifth stage.
High-Level Overview of the Stages and Phases of the Evaluation
Detailed planning of analysis stages
In actual fact, Figure 1 was created towards the end of the whole project, once the phases within each stage had been completed, to serve as a visual overview of the entire project.
At the outset of each stage, a plan was made for how to accomplish the analysis, or in the language of the Five-Level QDA method, how to translate the analytic tasks (strategies) into the use of software tools (tactics). Table 1. shows the initial high-level planning for the analysis of the client feedback.
Table 1. Stages of Analysis of the Client Feedback
Stages | Description of the analytic activities |
First Stage: Initial high-level analysis of Client feedback | With the broad aim of understanding aspects of the Programme that could be improved, this Stage focuses on the experiences of Clients, generating a high-level overview of satisfaction and drawing out specific areas that had been experienced less positively by Clients. The result will be a summary report providing a high-level overview and pulling out key areas of improvement. This will then be reviewed and the next stage planned. |
Second Stage: Exhaustive in-depth analysis of Client feedback | Building on the findings of the First Stage, the aim is to understand in-depth how the different aspects of the Programme had been experienced by Clients. The result will be a detailed report presenting the analysis of what had worked well . |
Stage 1 step-by-step documentation : initial high-level analysis of client feedback
Having planned at a high level, I began breaking down the objectives of the first stage of analysis and planning the detail of how to accomplish them. Whilst developing the Five-Level QDA method I experimented with several different ways to develop such plans, so they can be adapted as the analysis progresses, which is almost always what happens.
Table 2 shows the final version of what I did in the first stage of the analysis of the client feedback, organized according to phases, with each task numbered with how it was accomplished in MAXQDA in bullet points.
The chapter in the book describes these phases and illustrates how the software was used to accomplish them in a series of screenshots.
Table 2: Phases in the First Stage of the Analysis: Initial high-level analysis of Client feedback
Phase 1: Prepare and Clean Data |
1a. Retrieve all Client responses - download all available responses from both versions of the Client survey from the online survey tool into separate spreadsheets |
1b. Familiarize with and amalgamate responses - familiarize with the questions and responses in each spreadsheet - check question headers in Excel, align and combine into one spreadsheet - add a column containing a Unique ID for each respondent – i.e. R001 – R0082 |
Phase 2: Set-up MAXQDA project |
2a. Create a MAXQDA project - name the project ‘Pilot Client analysis’ and save in default location |
2b. Import survey data into MAXQDA The Survey Import routine (accessed from the Import Main Menu) was used to import data from the amalgamated spreadsheet, choosing the following options: - create a new Document Group for the responses - choose Unique ID column as the Document Name - import open-ended questions as Codes - import closed questions & demographic data as Variables - do not code empty cells |
2c. Check data and adjust for analysis Review the import process to check everything is as expected and adjust as follows: - rename the new Document Group to reflect its content (‘Client Survey Responses’) - create a top-level code called ‘Client OEQs’ and move all the codes created as part of the import process underneath - rename OEQ codes to shorter labels indicating content after checking each linked Code Memo contains the full questions asked |
2d. Begin documenting the analysis - Create a Memo called ‘Analysis Objectives’ and outline the purposes of the analysis - Start the Logbook, with the first entry summarizing the tasks undertaken so far |
Phase 3: Familiarize to inform analysis plan |
3a. Explore closed responses at a high level - using Document Variable statistics (Table and Chart Views) - create QTT Worksheet entitled ‘High Level Exploration’ to save Table and Chart views |
3b. Group less satisfied Clients - using the Activate by Document Variable feature, activate Documents from Clients giving a score of 7 or less and create a Document Set containing them |
3c. Explore open responses at a high level - using Word Frequency and Word Combinations – explore across all responses |
3d. Explore the relationship between closed and open questions at a high level - using Crosstabs for key open question Codes by key closed-question Variables |
Phase 4: Design initial analysis plan and prepare data |
4a. Plan the analysis steps - write intended analysis steps as a narrative in Project Memo - create a visualization of the analysis plan in a Map |
4b. Recode Variables for more focused analysis - Likelihood to Recommend & Month – Year recoded in Stats - NHS Trust sub-codes transformed into Document Variable |
4c. Isolate open-questions not being analyzed in this Stage - create separate top-level code and move open-ended response Codes not being analyzed |
Phase 5: Categorize responses to open-questions |
5a. Categorize reasons for not recommending the Programme - Reasons for not recommending the Programme amongst those less likely to do so were retrieved and categorized using the Categorize Survey Data tool and then visualized in a Map |
5b. Code all responses to each open question - starting with the ‘What could be improved’ open-ended question amongst those who were less likely to recommend the Programme, use the Categorize Survey Data tool to create sub-codes capturing Clients responses. - repeat for each open-ended response Code, in turn |
5c. Review and define categorized open responses - view sub-codes for each open-question, retrieving segments to review, merge codes where necessary - Generate brief definitions for each code at a linked Code Memo |
Phase 6: Prepare Summary Report |
6a. Generate visuals highlighting the key initial findings generate the following visuals and export into a PowerPoint presentation: - Document Variable statistic charts (showing sample breakdowns and likelihood of recommending the program) - Map showing retrieved segments for categorized reasons for lower recommendation score - Descriptive frequencies and qualitatively coded suggestions for improvements - Graphs and Code clouds of categorized responses to ‘What Worked Well’ and where impacts of the program were noticed |
6b. Generate newsletter for Coaches summarizing impacts - Export frequencies of categorized impacts codes to Excel to create hierarchy charts to visually represent impacts - Export Charts and coded segments to illustrate impacts qualitatively |
Stage 2 step-by-step documentation : in-depth analysis of client feedback
The second stage of analysis was designed after receiving feedback about the first report. The high-level analysis provided an important summary of the Programme from the Client perspective, a general sense of what had worked well, and a detailed understanding of potential areas for improvement from the perspective of those who were less likely to recommend the Programme to colleagues.
However, it was also clear that to learn lessons from the Programme that would be useful to other potential similar ventures in the future, more focus on the qualitative responses was required. The Second Stage therefore moved the emphasis of the analysis in that direction, and included all the feedback gathered from Clients between the start of the Programme to its conclusion.
Table 3 shows the final version of what I did in the second stage of the analysis of the client feedback, organized according to phases, with each task numbered with how it was accomplished in MAXQDA in bullet points.
The chapter in the book describes these phases and illustrates how the software was used to accomplish them in a series of screenshots.
Table 3 Phases in the Second Stage of the Analysis: Exhaustive in-depth analysis of Client feedback
Phase 7: Prepare and import additional data |
7a. Amalgamate additional responses - format Client responses collected since Phase 1 in Excel using same format as in Phase 1b. |
7b. Add new responses to existing data set - import additional responses into existing MAXQDA project - check new data has correctly aligned into existing structures |
Phase 8: Categorize additional responses into existing framework |
8a. Code all new responses - Using the same process as in Phase 5a. code all new responses using the Categorize Survey Data tool to existing sub-codes and/or newly created sub-codes as required - create a “quotable quotes” code to capture powerful expressions of impact |
8b. Re-review categorized open responses and definitions - view sub-codes for each open-question, retrieving segments to review, merge codes where necessary - alter existing definitions for each Code at its linked Code Memo to account for any changes resulting from newly coded responses |
8c. Summarize by open-ended question - create a QTT Worksheet for each open question - retrieve frequency summaries of the first open question responses using Sub-code Statistics - review Code Memos and summarize responses to the open question in the QTT Worksheet for that question - repeat for other open questions |
Phase 9: Identify Impacts and Experiences of the Programme |
9a. Reorganize Codebook into thematic categories - create a back-up of the MAXQDA project - drawing on Phase 8c. move all sub-codes to the top level of the Coding System – review, merge and move into themes that cut across the open questions |
9b. Interrogate patterns and relationships - create a color scheme for Charts that reflects the CtCaB branding - generate an overview of themes by creating Charts showing number and percentage of respondents mentioning categories within themes - break themes down by generating Tables of Themes/Categories by response frequency - explore relationship between Themes/Categories and NHS Trust (using Variable created in Phase 4b) using Interactive Quote Matrix generated from a Crosstab - identify and display qualitative responses by categorized theme – using a Map for each category and retrieving coded-segments within it |
Phase 10: Produce in-depth report of findings |
10a. Export visuals and narrative interpretation - all the QTT Worksheets created in Phase 8c and the Maps created in 9b. were exported from MAXQDA into MS Word - initial interpretations generated in Memos were also exported into MS Word and drawn upon to write the report - export frequency tables of Themes/Categories to MS Excel to create hierarchy charts |
10b. Write up final report - using the exports generated in Phase 10a. write up a 40 page final report |
The benefits of detailed planning
Tables 2 and 3 above are simplified versions of the Analytic Planning Worksheets that we developed to teach the skill of translating analytic strategies to software tactics. There are many resources around those worksheets available elsewhere on this website.
As researchers become increasingly familiar with the tools available in their choses qualitative software program (whether they’re using MAXQDA or another program), and as they become more adept at translation, the use of full Analytic Planning Worksheets becomes less and less necessary.
But I still plan all my computer-assisted qualitative analysis projects using the same principles, and I find tables such as shown here incredibly useful to manage the analysis as it evolves.
When we developed Analytic Planning Worksheets we intended them to be a teaching and learning tool, but many of our students and colleagues have used them to document their analysis as well, as I am doing here.
Qualitative researchers are not always very good at explicitly detailing how they go about their analysis. This is to the detriment of our profession. The use of dedicated digital tools to accomplish analysis enables us to concretely show what we did, which is difficult when using manual paper-and-pen methods or non dedicated tools like word processors or spreadsheets. In addition, documenting process using diagrams and tables such as those I used in this project can contribute to the important process of explicating how we do qualitative analysis.
More about the Five-Level QDA method
What is Five-Level QDA all about? Blogpost by Nick Woolf, May 2016
Three textbooks on the Five-Level QDA method, one each for ATLAS.ti, MAXQDA and NVivo, published by Routledge, 2017
Silver C, Bulloch S & Salmona M (2023) Integrating the online teaching of qualitative analysis methods and technologies: challenges, solutions and opportunities. In Nind M (ed.) Handbook of Teaching and Learning Social Research Methods. Edward Elgar Publishing
Silver C (2021) Tools for teaching computer assisted qualitative data analysis. MethodSpace article. Sage
Silver C & Woolf N (2021) Ensuring Analytic Strategies Drive the Use of Chosen QDAS program: The Five-Level QDA method. in Paulus T & Lester J Doing Qualitative Research in a Digital World. Sage
Silver C & Woolf N (2019) Case Study: Using the Five-Level QDA Method with Dedoose in Salmona, M., Lieber, E. and Kaczynski, D., Qualitative and Mixed Methods Data Analysis Using Dedoose: A Practical Approach for Research Across the Social Sciences. SAGE.
Silver C & Woolf N (2019) The Five-Level QDA Method. Foundation entry. Sage Methods Foundations.
Paulus TM, Pope EM, Woolf N & Silver C (2018): “It will be very helpful once I understand ATLAS.ti”: Teaching ATLAS.ti using the Five-Level QDA method, International Journal of Social Research Methodology