Qualitative research is not research at all?
Rant this way ~ Photo by Nesster, CC BY-SA
Heather pointed out these comments by Bob Garfield from a recent broadcast of On the Media (“Sentiment Analysis Reveals How the World is Feeling“):
I’ve been arguing for years that qualitative research, focus groups and the like, are not research at all. They don’t generate data. It’s statistically insignificant, easily manipulated, and from my perspective just as likely to be exactly wrong as exactly right.
Garfield then adds:
But it seems to me that what you’re dealing with is something that deals with all of my objections, because you’ve got the world’s largest focus group.
Sigh. This is wrong on so many levels, and anyone who is interested in ethnography already knows why, but just to touch on some of the problems:
- Qualitative research can generate data. The tweets used in Johann Bollen‘s [1] sentiment analysis (the subject of this OTM episode), interview transcripts, field notes, photos, audiorecordings, visual recordings: all data. Some research within the qualitative tradition also generates numeric data [2] by, for example, calculating measures of intercoder reliability, or in the analysis of card sorting tasks.
– - There is a lot more to statistical testing than statistical significance (and some controversy among statisticians about overuse of significance testing). There is also more to quantitative analysis than statistical testing. Bayesian inference, for example, could be thought of as quantitative analysis that is not necessarily statistical testing.
– - Similarly, qualitative research cannot be reduced to “focus groups and the like”. The purposes, strengths and weaknesses of focus groups are very different from those of other qualitative methods such as [participant-]observation and one-on-one interviews [3].
– - Using statistical testing as a marker for what is or is not research omits work that has formed the backbone of the sciences such as classical experimentation, disconfirmation by example, comparative methods for creating typologies and analyzing artifacts, etc.
– - “Easily manipulated”? Yup, research findings in general can be manipulated. Statistical testing is really easy to manipulate.
Garfield’s statement also suggests either ignorance or dismissal of mixed methods research, which, I would argue, is increasingly becoming a gold standard for research in some fields, such as public health.
There’s a hint at why mixed methods have become so important in public health research in Garfield’s comment about “the world’s largest focus group.” Bollen’s use of a large collection of corpora is well-suited to his purposes, but other purposes can require different or additional kinds of work.
Let’s say I do a giant public health survey. If a minority in my sample doesn’t interpret a word or phrase in the same way that the majority interprets it, if some questions make no sense at all from their perspective, if people writing the survey have no idea what minority members’ concerns or experiences even are much less how they’re relevant to health, then the survey results will be meaningless for that social group.
There is no such thing as a survey that is not culturally informed. Without ethnographic work and awareness, surveys, public health information and campaigns, etc., will likely be culturally informed by those who are most powerful and/or in the majority. Qualitative research is indispensable for addressing structural health inequities affecting the less powerful. Should ethnographic work focused on these inequities be patted on the head and assured that it’s nice, but it’s not-really-research? Fortunately, the NIH does not think so.
Sometimes I wonder if people miss how widespread and useful qualitative work is because it can be invisible (see Tricia‘s related post about the ‘Invisibility of Ethnography‘). A couple recent episodes of On the Media may clarify the kind of research that Garfield is dismissing here, while at the same time (perhaps unknowingly?) depending on it.
On Nov. 4th, Garfield spoke with social media researcher danah boyd about “Parents helping kids lie online.” The paper [4] behind this interview presents quantitative summaries of survey data — “real” research, perhaps, to Garfield. But hmm, how and why was this survey designed?
Recent Comments