top of page
Search

The crux of literature reviewing: structuring critical appraisals and using CAQDAS-packages

Updated: Nov 25, 2021


CAQDAS packages are incredibly useful tools for facilitating literature reviews. This post discusses an important, but infrequently discussed aspect of reviewing literature that can be facilitated by thoughtful and systematic use of CAQDAS-packages – developing and analysing critical appraisals of literature.

CAQDAS-tools for reviewing literature

Undertaking literature reviews using dedicated CAQDAS-packages is a well-discussed topic - after all, literature is a form of qualitative data, and reviewing is a form of analysis. As such, CAQDAS-packages include a range of tools that can be harnessed for the various tasks involved in reviewing literature – see another article I wrote for a brief overview.

Analysis of any form of qualitative data - for example, interview or focus-group transcripts, visual materials, open-ended responses to survey questions, social media content etc. - requires appropriate analytic strategies in the context of the broader methodology, and the choice of appropriate software tools to accomplish each individual analytic task. There is therefore no one process suitable for all qualitative data analysis projects. Literature reviewing is no different.


Strategies drive tactics

Choosing the appropriate tools within your chosen CAQDAS-package to undertake analytic tasks depends on the review methodology. This is what we mean when we say strategies (what you plan to do) drive tactics (how you plan to do it). For example, different tools will be appropriate when undertaking a systematic literature review (e.g. a meta-analysis or meta-synthesis), an in-depth literature review, an evidence review, a critical review, a State-of-the-Art review or any other type of literature review. See Grant & Booth (2009) for an overview of different review methodologies


Dedicated CAQDAS-packages provide many tools that we, as researchers and analysts, pick and choose for our analytic needs. Different tools will be appropriate to use at different stages of a literature review, and for different purposes and in different ways, depending on the needs of the review. There is no one recipe – but CAQDAS-packages are very flexible, so we have many options.


Critically appraising the literature is the crux of the process

What is common across all types of literature review is the need to appraise what is ‘going on’ in the literature in relation to the objectives of the review, and writing a report, chapter, article or thesis section as the output. I’ve seen lots of examples of literature review projects that haven’t been properly planned, where for example, researchers have launched into coding full-text PDF articles within their chosen CAQDAS-package without analytic focus in the moment, or a sense of direction moving forwards. Whatever the type or focus of a literature review, they all require critical reading and critical writing.


Implementing critical reading and writing from the outset

Implementing a critical reading and writing approach from the outset is a powerful tactic that cross-cuts all forms of literature reviewing. Directly coding full-text PDF literature articles may well be an important part of the literature reviewing process, but it needs to be done in context and with focus. Thinking about what you’re reviewing, why, and how – and developing systematic structures for capturing those thoughts - is always the best starting point.


Clarity around review purpose

Whatever the type of literature review, it’s crucial to be clear about the overarching purpose. Systematic literature reviews seek to explicitly answer specific questions, which drive every stage of the process - from developing inclusion/exclusion criteria for identifying relevant literature, through the framing of review questions, identifying the focus of analysis and the structuring of the reporting.


When working more exploratively, for example in an in-depth literature review seeking to identify gaps in scientific knowledge and develop research questions, the overall purpose and the development of specific review questions is also key. Just because exploratory work is iterative and emergent, doesn’t mean that we don’t need to be focused or systematic in our processes.


General questions for critical reading and appraising (Wray & Wallace)


Critical Reading and Writing for Postgraduates by Mike Wallace and Amanda Wray

One of my favourite books on this topic is Critical Reading and Writing for Postgraduates by Mike Wallace and Amanda Wray. If you’re not a postgraduate researcher don’t be put off by the title – this book is well worth reading whatever stage of your career or sector you work in, and whatever type of literature review you’re undertaking. They suggest a series of questions to ask yourself when reading literature to critically analyse texts and develop structured synopses.









For example, in developing a critical synopsis of a text, they suggest asking the following questions:

  1. Why am I reading this?

  2. What are the authors trying to achieve in writing this?

  3. What are the authors claiming that is relevant to my work?

  4. How convincing are these claims, and why?

  5. In conclusion, what use can I make of this?


And in critically analysing a text, they suggest asking the following questions

  1. What review question am I asking of this text?

  2. How and why are the authors making this contribution?

  3. What is being claimed that is relevant to answering my review question?

  4. How certain and generalized are the authors’ claims?

  5. How adequate is the backing for these claims?

  6. How effectively does any theoretical orientation link with these claims?

  7. To what extent does any value stance adopted affect claims?

  8. To what extent are claims supported or challenged by others’ work?

  9. To what extent are claims consistent with my experience?

  10. What is my summary evaluation of the text in relation to my review question?

Substantive questions to focus appraisals

In addition, to fine-tune your critical mind when reading and appraising literature, it’s useful to think about the substantive areas of interest that you need to identify and critique as well. Having developed a clear purpose, this should be relatively straight-forward to do – although of course the more explorative the review, the looser, higher-level, and generic the substantive questions may be. Taking the time during the planning phase to do this will pay dividends later on. It’s highly likely in more explorative reviews that these substantive questions will change as you appraise more pieces of literature. But they serve to focus your reading and the appraisals you develop around your purposes.


Here you can see an example of substantive review topics from a literature review Nick and I undertook, in which we developed a clear purpose and set of 9 topics to focus on.


Developing a literature review template


Developing a review template

It’s important to spend time developing an empty template containing your initial review questions. On the companion website for Critical Reading and Writing for Postgraduates by Mike Wallace and Amanda Wray you can download their Critical Analysis and Critical Synopsis templates – which you can adapt, and add your substantive questions to.

Most CAQDAS-packages have a note-taking or memo feature where this template can be created, providing a central place to write the review purpose and develop a template of critical and substantive questions. If not, you can do this in an MSWord file (or equivalent) outside of your chosen CAQDAS-package. Either way, once developed, the template can be copy/pasted for each appraisal you write. This ensures you take equivalent notes about each piece of literature, and puts in place, from the outset, the basis for analysing your appraisals later.


Setting up the ability to analyse your critical appraisals

Developing critical appraisals will likely involve several rounds of reading and appraising. It might be, for example, that you initially skim-read each piece of literature – perhaps to answer Wallace & Wray’s initial questions – and then you revisit the literature to focus on your more substantive areas of interest, appending each critical appraisal accordingly. But at some point, you’ll need to analyse your critical appraisals.


This means you’ll need to give each critical appraisal the ‘status’ of a data file in your chosen CAQDAS-package. Some CAQDAS-packages allow you to write your critical appraisals directly into analysable data files (usually called ‘documents’). In others, you may need to first write the appraisals using the note-taking or memo feature, and then convert them into analysable data files.


The point is that putting thought into what you’re reading the literature for, will help ensure you take equivalent notes about each piece of literature and thereby focus your appraisals around review objectives. Those review sections can then be coded, allowing you to later retrieve, compare and analyse your notes about each review section across all pieces of literature – or across sub-sets.


Three types of coding

It’s useful to consider the following three types of coding when reviewing literature

- Structural coding of critical review sections

- Substantive / thematic coding of appraisals (and maybe also full-text PDFs)

- Literature reviewing coding


Structural critical review sections

Having used your template to write your critical appraisals, it will be possible to auto-code each review section across all your critical appraisals quickly within your chosen CAQDAS-package. The result will be a code for each review section, at which all your notes about that question, across all your critical appraisals, are coded. This allows you to easily compare your notes on each review question across all your reviews, or indeed amongst a sub-set of reviews.


Reviewing your earlier notes by each review question together in one place like this, allows you to start thinking - and writing - at a higher level and to make some comparative evaluations about sub-sets of the literature.


Thematic or substantive coding of critical appraisals and literature files

When reviewing your critical appraisals, you will likely see some recurring substantive topics and themes – indeed you may already have a potential list of these derived from clear thinking about the literature review purpose (see above). These codes can be organised in the CAQDAS-package coding scheme, and applied across your critical appraisals and any full-text articles you have imported into the project. Just like when coding other qualitative materials, some of these codes may be developed deductively, and others inductively. The software doesn’t care – although your review methodology probably does!


Literature reviewing codes

When taking a critical reading/writing approach to appraising literature, it’s also almost certainly useful to use what I call ‘literature critique codes’. What these are, their definitions and how you use them is up to you, and as always, grounded in your review objectives and methodology. Some examples include:

  • · Refs to get

  • · Un-substantiated claims

  • · Quotable quotes

  • · Facts to find

  • · Refs to include in write-up

Just like substantive/thematic codes, these literature reviewing codes can be applied across critical appraisal sections and directly on the literature files themselves. That allows you to integrate the actual literature and your appraisals of it … subject of my next blogpost…

References

bottom of page