Qualitative research is not research at all?

Rant this way ~ Photo by Nesster, CC BY-SA

Heather pointed out these comments by Bob Garfield from a recent broadcast of On the Media (“Sentiment Analysis Reveals How the World is Feeling“):

I’ve been arguing for years that qualitative research, focus groups and the like, are not research at all. They don’t generate data. It’s statistically insignificant, easily manipulated, and from my perspective just as likely to be exactly wrong as exactly right.

Garfield then adds:

But it seems to me that what you’re dealing with is something that deals with all of my objections, because you’ve got the world’s largest focus group.

Sigh. This is wrong on so many levels, and anyone who is interested in ethnography already knows why, but just to touch on some of the problems:

• Qualitative research can generate data. The tweets used in Johann Bollen‘s [1] sentiment analysis (the subject of this OTM episode), interview transcripts, field notes, photos, audiorecordings, visual recordings: all data. Some research within the qualitative tradition also generates numeric data [2] by, for example, calculating measures of intercoder reliability, or in the analysis of card sorting tasks.
  –
• There is a lot more to statistical testing than statistical significance (and some controversy among statisticians about overuse of significance testing). There is also more to quantitative analysis than statistical testing. Bayesian inference, for example, could be thought of as quantitative analysis that is not necessarily statistical testing.
  –
• Similarly, qualitative research cannot be reduced to “focus groups and the like”. The purposes, strengths and weaknesses of focus groups are very different from those of other qualitative methods such as [participant-]observation and one-on-one interviews [3].
  –
• Using statistical testing as a marker for what is or is not research omits work that has formed the backbone of the sciences such as classical experimentation, disconfirmation by example, comparative methods for creating typologies and analyzing artifacts, etc.
  –
• “Easily manipulated”? Yup, research findings in general can be manipulated. Statistical testing is really easy to manipulate.

Garfield’s statement also suggests either ignorance or dismissal of mixed methods research, which, I would argue, is increasingly becoming a gold standard for research in some fields, such as public health.

There’s a hint at why mixed methods have become so important in public health research in Garfield’s comment about “the world’s largest focus group.” Bollen’s use of a large collection of corpora is well-suited to his purposes, but other purposes can require different or additional kinds of work.

Let’s say I do a giant public health survey. If a minority in my sample doesn’t interpret a word or phrase in the same way that the majority interprets it, if some questions make no sense at all from their perspective, if people writing the survey have no idea what minority members’ concerns or experiences even are much less how they’re relevant to health, then the survey results will be meaningless for that social group.

There is no such thing as a survey that is not culturally informed. Without ethnographic work and awareness, surveys, public health information and campaigns, etc., will likely be culturally informed by those who are most powerful and/or in the majority. Qualitative research is indispensable for addressing structural health inequities affecting the less powerful. Should ethnographic work focused on these inequities be patted on the head and assured that it’s nice, but it’s not-really-research? Fortunately, the NIH does not think so.

Sometimes I wonder if people miss how widespread and useful qualitative work is because it can be invisible (see Tricia‘s related post about the ‘Invisibility of Ethnography‘). A couple recent episodes of On the Media may clarify the kind of research that Garfield is dismissing here, while at the same time (perhaps unknowingly?) depending on it.

On Nov. 4th, Garfield spoke with social media researcher danah boyd about “Parents helping kids lie online.” The paper [4] behind this interview presents quantitative summaries of survey data — “real” research, perhaps, to Garfield.  But hmm, how and why was this survey designed?

On the basis of discussions with parents prior to conducting our survey, we learned that parents did not identify COPPA or even the general issue of privacy as the source of age restrictions on Facebook. Instead, they often told us that Facebook, MySpace, and Twitter are “mature” sites, meant only for teenagers who are old enough and mature enough to handle the various online safety issues present. Meanwhile, many of the same parents appeared to know that their under–13 children were accessing various social media sites. Given our ongoing conversations with parents in other research contexts and the high incidence of children lying about their age to gain access to sites that purport to forbid them, we sought to investigate whether children are evading age restrictions against their parents’ wishes, whether parents are agnostic or unaware of such restrictions, or whether parents are complicit in children’s covert participation on these sites.

The discussions mentioned turn out to be “semi-structured interviews”, i.e., qualitative research.  The development and motivation for the survey is informed by both qualitative information  (“ongoing conversations with parents in other research contexts”) and quantitative information (“the high incidence of children lying about their age to gain access to sites that purport to forbid them”).

The discussion section of the paper follows up on how parents think about age restrictions:

In general, parents do not understand the relationship between Facebook’s minimum age requirement and COPPA… Given the frequency with which parents who knew there was a minimum age referenced issues of age appropriateness or maturity in an open–ended question — and given explanations we heard in qualitative work done as a pilot for this study — one explanation may be that parents see age restrictions as a form of a maturity restriction or a type of maturity rating.

A great deal of Garfield’s interview with boyd centers on this narrative about the disconnect between the motivations for COPPA and parents’ interpretations of age restrictions. It is, after all, one of the most interesting and meaningful parts of the paper. Without qualitative research, the follow-up survey would not exist, and the overall meaning and relevance of the research would be lost.

Sentiment analysis can’t do anything remotely equivalent to what qualitative research accomplished here, nor can other NLP tools. Which is not to say that there aren’t zillions of possibilities for NLP-based analysis of, say, parents’ and children’s comments on Facebook that could be useful, but rather that computational models cannot perform the same kinds of investigation, synthesis and analysis (What do you call that stuff? Oh yeah, research!) that people can do.

So mixed methods research is important, and is actually research.

In another On the Media broadcast, Garfield interviews the ethnographer Gabriella Coleman [5] about the hacker group(s) Anonymous. Coleman says:

Within each network there are certain participants who can allow or disallow certain people. And what happens then is that people go elsewhere and set up their own network. Within a network there are techniques that are used to, quote, unquote, “keep people in line.” When they were engaging in operations in Tunisia, some people were talking about attacking the media. And when you have a swell of people saying, no, no, that’s not what we do, it kind of keeps people in line on that network. So there are forms of control, but to be sure, as you noted, anyone can then take the name and as long as they corral enough human power, they, too, can lead actions as well.

I’m pretty sure that Coleman did not learn this, or most of the other valuable information she provides in the interview, by doing automated sentiment analysis.  She does, however, seem to have done research, likely involving close readings of group members’ words and actions, observation, and/or other qualitative methods.

At the end of the interview, Garfield asks Coleman:

Anonymous acts in the name of good government and justice and transparency in ways that are unilateral, sometimes arbitrary and certainly opaque. How does Anonymous justify that, you know, central irony, maybe even hypocrisy?

Good question. If Garfield were to frame it as a research question, I bet he would need to incorporate qualitative methods to answer it.

Coleman responds:

There are some participants who don’t engage in the kind of lulzy trollish behavior, and so for them there is no contradiction. What they do is civil disobedience.

It’s an off the cuff response, not an excerpt from a paper, but in any case I could disagree with Coleman  (have no reason to, but just saying) — or with any qualitative or ethnographic analysis. I could think a finding or discussion point is missing necessary information, or its claims are overly broad, or that it conflates unrelated topics.

But, I could think that about any kind of research.  Addressing a question like Garfield’s as a research subject without qualitative methods, would likely be ineffective, inefficient and/or foolish. No method inherently produces “exactly right” results. Only researchers who understand the applications and limitations of the methods they are using, who have insight into their data, and who think rigorously, can produce good research.

And yes, “research” includes qualitative research. Why do we only hear a journalist’s misinformed viewpoint on this broadcast, with no mention of the mainstream viewpoints of actual researchers?

—

______________
References

Translation

English中文(简体)FrançaisEspañol

by

You can help improve the automatic translation by checking off “Edit Translation” after you select a language. Thanks!