A key principle of the Five-Level QDA(R) method is that analytic strategies drive software tactics. This ensures that software use is always focused on the objectives of the study and is appropriate to the particular methodological context.
But what happens when software tactics drive analytic strategies? Here is the most extreme example I’ve observed. In my next post I’ll discuss when software tactics can usefully inform analytic strategies without actually driving the process.
Cool CAQDAS features
Dedicated CAQDAS packages now have many new “cool” features. This is great. Many ways of working with qualitative and mixed data are now practical that used to be difficult or impossible, extending the way we think about analytic possibilities.
Take literature reviews. New and better tools in various CAQDAS packages allow more effective analysis of bodies of literature. For example:
References exported from bibliographic software (such as Endnote, RefWorks, Zotero and Mendeley) can be imported (often along with the full-text PDF articles and associated metadata)
Search tools can identify, visualise and capture frequently occurring words and phrases that represent concepts and theories - and the surrounding context (e.g. sentences, paragraphs etc.)
Appraisals written about the literature being read can themselves be analysed
Mapping and other visualisation tools can be used to display associations between and within the concepts identified in literature
New interrogation tools can be used to compare analysed literature in a variety of different ways
What can go wrong: an extreme horror story
A participant in one of my workshops had been using the software for a number of months for her literature review because she’d heard it was a useful thing to do, but she couldn’t “see how it was helping her”. This was her primary reason for attending the workshop. I asked what she’d done and why she felt it wasn’t working for her, and she showed me around her software project.
650 full-text PDFs
She’d imported 650 full-text PDF articles using the program’s cool bibliographic import feature. The import had done several cool things, including linking all the meta-information about the articles from the bibliographic software to the PDFs. No problem with that - importing in this way is quick and easy and useful in many circumstances. The problem lay in what she’d done next.
Hundreds and hundreds of codes
Her CAQDAS project now contained several hundred codes. That’s not necessarily a problem either - sometimes hundreds of codes are entirely warranted. Each code was linked to several hundred - and frequently thousands - of data segments. Again, not necessarily a problem. If you’re working with 650 articles then each code may well need to be linked to many segments of data, because there’s so much text to review.
Warning bells
Looking at the long list of codes alerted me to why she wasn’t finding the software was “helping with the literature review”. Each code was named with only one word and none of the codes were defined.
Using codes within a CAQDAS package is a tactic, not a strategy, and these codes can be used for a variety of purposes. The commonly intuitive way is to use them to represent concepts within the data, and they typically have a two or three word codename to name the concept.
There are many circumstances when codes are appropriately used for other purposes, but when every code is named with just one word and none of them have been defined, they appear not to be representing concepts - and I wonder why.
The problem
The problem turned out to be that all of these codes and the data segments linked to them had been created using keyword-based auto-coding features without reading the articles. She had found out about the software’s word frequency and auto-coding features and used them to identify frequently occurring words and then coded every sentence in which each seemingly interesting word occurred without reading anything.
Using “cool” features without purpose
The process at the tactics level had been pretty quick - run the word frequency feature, choose a word, and tell the software to code each ‘hit’ to a code whose name is the word being searched, together with the sentence around the hit. Boom!
I’m not criticising word frequency or auto-coding features in CAQDAS packages. They’re very powerful and invaluable in many circumstances. But only when they’re used with purpose. This researcher had been seduced by a “cool” feature and used it without focus - in other words, without thinking about how the resulting coding would help her, or what she’d do next. This is because the software tactics came first, and effectively became the analytic strategy which proved not to move the analysis forward.
CAQDAS packages can’t interpret
Using the cool feature without purpose wasn’t the only problem - she’d also expected that the use of the software would be a short-cut to getting the literature review done without needing to actually read the articles. She expected that capturing every sentence in which frequently occurring and seemingly interesting words occurred was somehow going to “tell” her what was going on in the articles. She’d been seduced into thinking that the software could tell her something about the literature without her having to do the work of reading and thinking.
This is an extreme example of misunderstanding the role of CAQDAS packages in the analytic process. It’s true that some tasks can be undertaken more quickly and reliably in CAQDAS packages than in non-dedicated software or when using manual methods. But interpretation isn’t one of them.
Strategies first
The use of CAQDAS packages should be driven by analytic strategies, not by its “cool” features. The features provide a range of tools that we pick and choose to fulfil our analytic tasks, and the software’s power lies in this flexibility.
But just because something is possible doesn’t mean it’s sensible.
CAQDAS features can certainly be harnessed to facilitate literature reviews - and some have been specifically designed for the purpose. But there are different types of literature reviews, and a range of strategies for accomplishing them. That’s why CAQDAS programs have so many features. We don’t need them all for each individual literature review or data analysis. Letting the software drive the process will often be problematic. The use of features must be driven by the needs of the analysis, not just because they’re available and seem “cool”.
CAQDAS features can inform our strategies, just don't let them drive
Of course making sure that the software doesn’t drive the process doesn’t mean that the capabilities of the software are irrelevant. In the next post I’ll discuss when software tactics can usefully inform analytic strategies, without crossing the line of driving the analysis.