Archive | November, 2011

Qualitative research is not research at all?


Image of building with torn sign reading "Rant"

Rant this way ~ Photo by Nesster, CC BY-SA

Heather pointed out these comments by Bob Garfield from a recent broadcast of On the Media (“Sentiment Analysis Reveals How the World is Feeling“):

I’ve been arguing for years that qualitative research, focus groups and the like, are not research at all. They don’t generate data. It’s statistically insignificant, easily manipulated, and from my perspective just as likely to be exactly wrong as exactly right.

Garfield then adds:

But it seems to me that what you’re dealing with is something that deals with all of my objections, because you’ve got the world’s largest focus group.

Sigh. This is wrong on so many levels, and anyone who is interested in ethnography already knows why, but just to touch on some of the problems:

  • Qualitative research can generate data. The tweets used in Johann Bollen‘s [1] sentiment analysis (the subject of this OTM episode), interview transcripts, field notes, photos, audiorecordings, visual recordings: all data. Some research within the qualitative tradition also generates numeric data [2] by, for example, calculating measures of intercoder reliability, or in the analysis of card sorting tasks.
  • There is a lot more to statistical testing than statistical significance (and some controversy among statisticians about overuse of significance testing). There is also more to quantitative analysis than statistical testing. Bayesian inference, for example, could be thought of as quantitative analysis that is not necessarily statistical testing.
  • Similarly, qualitative research cannot be reduced to “focus groups and the like”. The purposes, strengths and weaknesses of focus groups are very different from those of other qualitative methods such as [participant-]observation and one-on-one interviews [3].
  • Using statistical testing as a marker for what is or is not research omits work that has formed the backbone of the sciences such as classical experimentation, disconfirmation by example, comparative methods for creating typologies and analyzing artifacts, etc.
  • “Easily manipulated”? Yup, research findings in general can be manipulated. Statistical testing is really easy to manipulate.

Garfield’s statement also suggests either ignorance or dismissal of mixed methods research, which, I would argue, is increasingly becoming a gold standard for research in some fields, such as public health.

There’s a hint at why mixed methods have become so important in public health research in Garfield’s comment about “the world’s largest focus group.” Bollen’s use of a large collection of corpora is well-suited to his purposes, but other purposes can require different or additional kinds of work.

Let’s say I do a giant public health survey. If a minority in my sample doesn’t interpret a word or phrase in the same way that the majority interprets it, if some questions make no sense at all from their perspective, if people writing the survey have no idea what minority members’ concerns or experiences even are much less how they’re relevant to health, then the survey results will be meaningless for that social group.

There is no such thing as a survey that is not culturally informed. Without ethnographic work and awareness, surveys, public health information and campaigns, etc., will likely be culturally informed by those who are most powerful and/or in the majority. Qualitative research is indispensable for addressing structural health inequities affecting the less powerful. Should ethnographic work focused on these inequities be patted on the head and assured that it’s nice, but it’s not-really-research? Fortunately, the NIH does not think so.

Sometimes I wonder if people miss how widespread and useful qualitative work is because it can be invisible (see Tricia‘s related post about the ‘Invisibility of Ethnography‘). A couple recent episodes of On the Media may clarify the kind of research that Garfield is dismissing here, while at the same time (perhaps unknowingly?) depending on it.

On Nov. 4th, Garfield spoke with social media researcher danah boyd about “Parents helping kids lie online.” The paper [4] behind this interview presents quantitative summaries of survey data — “real” research, perhaps, to Garfield.  But hmm, how and why was this survey designed?

Read More… Qualitative research is not research at all?

Data conversations: Can ethnographers do numbers?


I was grappling for a long time about how to approach numbers, statistics, patterns in my Wikipedia research. I’m an ethnographer, right, we’re supposed to be averse to using numbers. Right?

And then Rachelle sent me this really interesting piece by Ken Anderson and the crew from Intel’s People and Practices Research group called ‘Numbers Have Qualities Too: Experiences with Ethno-Mining’ (Ken Anderson et al, 2009) [PDF].

And I realised that there is no problem with numbers and statistics per se. The problem is when we use numbers divorced from the context in which they are extracted. The problem comes when we use numbers to speak for a community, rather than enabling the community to speak to the numbers.Read More… Data conversations: Can ethnographers do numbers?

Technology Unfolds Over Time


Ghana Internet Cafe pic by Rachel Strohm CC BY ND 2.0

Many ethnographers stick to one place or region throughout their careers. Perhaps the memory of the trials and travails of entering the field in the first place, of early incomprehension and discomforts, the exhaustion of language learning, makes them shudder to imagine starting that all over again. Social ties to the field can be maintained in new ways (such as through Facebook). For example. Over time these relationships become deeper, richer. It becomes easier to ask more sensitive and private questions. One develops a growing capacity for insight into a culture, for the non-public side of society, for a better understanding of social performances vs. personal idiosyncrasies, the cleavage points beyond a society’s well-ordered face.

After 7 years of traveling to Ghana, I’ve started to see this sense of time and of change emerge in my own work as well. In part experiencing this more private side of life, but also observing firsthand the changes made sharper and more apparent by my absences. The Internet café scene in Ghana is not what it was when I started fieldwork in 2004. It was around 2008 that I started to see reports from the news media and people in Ghana about ‘sakawa’ a vernacular term that referred to Internet fraud. This was a term only whispered about during my fieldwork but had emerged around 2008 as part of a very public moral debate and was incorporated into the narratives of Ghana’s popular culture – in music and local video-films.

To formalize this sense of passing time I re-interviewed 12 individuals from my 2004 fieldwork. I believe this makes my study the very first longitudinal examination of the Internet in Africa. I was especially interested in whether, with time, the Internet yielded benefits to this group, delivered on their initial enthusiasm and conviction in the way the Internet worked (which in 2004 was bolstered by astonishing second-hand stories/rumors of big gains but very little successful direct experience among users).Read More… Technology Unfolds Over Time

IDEO.org Opens 2012-2013 Call for Fellows


IDEO is opening it’s call for fellows at Ideo.org. This seems like a great opportunity for ethnographers to apply!

IDEO.org has issued its second annual call for fellows to serve as “innovators in residence.” The 2013 class of fellows will follow eight diverse fellows, currently part of the nonprofit start-up’s inaugural year. Those eight were selected from an applicant pool in 2011 of over 400. Phase 1 applications for the 2012-2013 fellowship year are due Friday, December 9, 2011; select candidates will be invited to participate in a more in-depth round of reviews, followed by interviews.

The IDEO.org Fellowship Program enables future global leaders from the design, business, and social sectors to spend one year working with experienced IDEO designers to address poverty-related challenges using the tools of human-centered design. Over a 12-month period, Fellows will deliver solutions for non-profits, social enterprises, and foundations on an array of topics, such as: agriculture, gender equity, financial services, health, water, and sanitation.

Click here to learn more about IDEO.org and its fellowship program.

The Invisibility of Ethnography


What are ethnography’s doings? I mean, really, how do you describe what exactly an ethnographer does? S/he watches people? Explains people’s feelings? Translates cultural ideas into concrete stuff? I’ve come up with some interesting ways that work for me to describe my work, but it still requires context and to a person who has never worked with an ethnographer before, it’s not always clear.

Heller Communication writes about the invisibility of socially innovative design.

Design for social innovation begins with the design of conversations themselves – it requires treating a conversation with the same care, and the same planning, that would be appropriate for the design of a product. Conversation starts everything – and yet we rarely think of them as an opportunity for design. This is not only the most important, upstream part of the systems that we need to change, it’s the fastest way for a designer to become a vital part of a strategic initiative. It’s where things begin, and where the most important things are decided.

On the hard side, it doesn’t provide much of a portfolio. Nothing to enter into design competitions, few samples to put on your website, harder to explain at a cocktail party just what it is that you do. In fact, most of the invisible things you’ll be designing are private and sensitive to CEOs and leaders of all types of organizations. You can’t even talk about them. This can be a tough shift for designers who are loathe to give up the artifacts of their work. Of course it doesn’t mean that you won’t design any artifacts, it only means that they will be the last thing you design, not the first.Read More… The Invisibility of Ethnography

Welcome to Oscar Grant Plaza


A couple weeks ago I woke up at five in the morning to what sounded like a battalion of helicopters overhead. It was not the first time. Whenever there’s been a protest in my downtown/uptown Oakland neighborhood following a new development in the Oscar Grant case, out come the helicopters and police.

I figured it was Occupy Oakland being raided since there had been rumors the police would come early in the morning, and I went outside to look around. The streets were barricaded for blocks, and there was no way to see what was going on inside.

When I returned later, the plaza was still barricaded and guarded by a line of police in riot gear. Occupy Oakland protesters were amassed outside the barricades, some sitting on the sidewalk with backpacks and sleeping bags. I wondered if they were planning to move back in, if they had somewhere else to go, and how they saw the space of the plaza they had inhabited. I went home and came back with some markers and paper, hoping that some protesters would be interested in drawing pictures of Oscar Grant Plaza (the name Occupy Oakland gave to Frank Ogawa Plaza when they moved in) or maybe of Oscar Grant — something to capture the place they had created.

The drawings people did of Oscar Grant Plaza, especially, got me thinking about place and space in the sense that Harrison & Dourish describe in this piece (pdf)[1]. In their terms, space is an opportunity or collection of affordances. Place, meanwhile is:

generally a space with something added—social meaning, convention, cultural understandings about role, function and nature and so on. The sense of place transforms the space. (p. 3)

The place Oscar Grant Plaza was before the raid — a space with something added — looked like this to one of the Occupy Oakland protesters, Luka:

Drawing of the Occupy Oakland encampment Oscar Grant Plaza before the raid

Before Police @ Oscar Grant Plaza ~ Drawing by Luka

Read More… Welcome to Oscar Grant Plaza