Judd and Tamar
Editor’s Note: Judd Antin @juddantin is a social psychologist and user experience researcher who studies motivations for online participation. In 2011, he was named an MIT Technology Review Innovator Under 35. Prior to joining Facebook, he worked with Yahoo Research. His educational background includes Applied Anthropology, Information Science, and training at the French Culinary Institute. One of my favorite papers of his is Readers are Not Free Riders: Reading as a form of participation on Wikpedia (pdf) [1].
Tamar Antin is a research scientist who uses mixed and especially qualitative methods to critically examine public health policies and narratives. She has several years of experience in public health research. One of her recent publications is Food Choice As a Multidimensional Experience [2]. Her dissertation [3] combining three papers on food choices and body image is excellent reading for any student of qualitative methods.
I’ve known Tamar and Judd for several years now, and Tamar has been a mentor to me. Every time Tamar and I talk about research and ethnography, it never seems to last long enough; I just want to ask her more questions. And every time I see Judd, I want to ask him a million questions too. So a post for Ethnography Matters was a great excuse to get together with them for a chat on anthropology, Big Data and Small Data, and other interesting things. – Rachelle
P.S. This isn’t a straight transcript of our conversation but a sort of Frankenstein transcript made out of chopped up pieces sewn back together.
_____________________________________________________________________
1. Two Ethnographers
2. What they’re working on
3. Stigma and hacking
4. Qualitative research as art, science and handmaiden
5. Big Data and Small Data
1. Two Ethnographers
What’s your background in anthropology?.
Judd: I have an undergraduate degree in anthro from Johns Hopkins, where I was one of seven anthropology majors I think, like in the whole university. It was a small department. I got interested in anthro primarily because of my adviser, who became our friend, Felicity Northcott. Coincidentally she also married Tamar and I. She was internet ordained and she officiated our wedding. She’s awesome. She was just a very down to earth, foul-mouthed, passionate anthropologist.
Tamar: And for me, I have an undergraduate degree in anthropology also, from the University of Texas. I was having this conversation with the undergraduate adviser there at the end of my senior year, like okay now I have this degree, but I didn’t really know what to do with it. I went to the career center, and they had a list of all the jobs that you could do with certain majors, and I think the only job that was listed for anthropology majors was travel agent.
Judd: What?
Tamar: Oh yeah. I was thinking, well I don’t want to do that.
Judd: Travel agent?!
Tamar: Yeah. So he encouraged me to go into Applied Anthropology, because he asked me some of the things that I was interested in, and so I applied for the Applied Anthropology program at Maryland.
Judd: It was a very practical program. We had a great cohort of people, and we got involved in a bunch of stuff and we learned a lot.
Tamar: Our adviser was Tony Whitehead, who actually was instrumental in both of our careers. He had been contacted by David Fetterman, who is an anthropologist at Stanford who was doing an empowerment evaluation that was funded by Hewlett Packard. He thought Judd would be great being involved in it, and it was a project related to the digital divide.
And then I had been approached by a friend of mine who was doing a project with a nutritionist on food access, and they wanted somebody to do an ethnography. Tony had been one of the first people to write about food and public health. He got me connected to some other projects later, and so I think he really shaped both of us in terms of the directions that we took after grad school.
And you guys met each other there, right?
Tamar: Mm hmm. We did.
(Tamar is totally onto my ‘Can you say more about that?’ pause.)
Tamar: …Do you want to hear about that?
Sure.
Judd: She just doesn’t want to say no.
No, I do. It’s actually on my paper, but it seems like a hokey question to just ask.
Tamar: See, this is like the love story of the two ethnographers, right?
Tell me the love story of the two ethnographers.
Tamar: It’s very short. We met at orientation, and became friends immediately. And within three weeks, let’s say a month, we were living together.
Judd: It happens.
Tamar: It was very fast.
Judd: When you’re an anthropologist right, you have to be very perceptive.
(Tamar laughs)
Judd: You have to be able to understand other points of view, and be empathetic, and so…
Tamar: And what does that have to do with anything?
Judd: It has to do with the fact that you notice other people really quickly! You evaluate them, whether they’re horrible or not, and you can tell pretty quickly ‘Yup, no problem, let’s move in.’
Tamar: That’s total bullshit!
(A conversation on the use of anthropological skills in relationships ensues, but is omitted here…)
2. What they’re working on
So what kind of stuff are you doing now?
Judd: What I do, is I work for Facebook. I would say that the work that I do is no longer very anthropological. But I have always said that even as I moved further away, toward social psychology, toward information science, that I credit anthropology and specifically developing an anthropological perspective with a lot of the way I think about research. Because – Tamar can attest to this because I think she finds it annoying sometimes – a lot of the things I do, I often skip to the ways in which people in contexts are different. When you understand the different frames of people as they are in online social media or online collaborative systems – I don’t know if you want to call it culture with a capital C or a lowercase C, but it matters to me a lot.
Recently I looked at motivations for doing work on Mechanical Turk, and one of the things that was most fascinating about it was the cross-national aspects. We did it in India and the United States. I would argue that talking about culture at the national level is not particularly useful, but I’m interested in those comparisons, and I’m interested in the way that cultural perspectives shape people and their interactions through technology. So that definitely is inside all of the work that I do, even if it’s not strictly anthropological.
Is that coming through in some of the work you’re doing now at Facebook?
Judd: Yeah. And there is one example that I think is not surprising. Most people who use Facebook do not live in the United States, and yet here we are in Silicon Valley, and we are working pretty hard to understand the perspectives of people who are getting on Facebook in Nigeria and Indonesia, in Vietnam, and Russia. We have hundreds of millions of people in these places. And so recently people on my team did this almost ethnographic trip where they went to a bunch of different countries, trying to understand the environment there as it related to the use of social media, and basic phones, and the technical infrastructure, and the social conventions and norms. I think that kind of work is going to become ever more important. If you believe that culture is important to the way that people use technology and that it should be baked in, and that you can’t form assumptions based only on this ethnocentric point of view, then I think you have to be an anthropologist. You have to be interested in cultural differences and frames of reference, and how they relate to technology use.
What are you working on these days Tamar?
Tamar: I work at public health research institutes in the Bay Area. I would describe myself as a critical public health researcher. The critical part comes from my training in anthropology, I would say. I definitely self identify as an anthropologist and not a public health researcher, because of the type of research that I do. The two proposals that I’ve been working on most recently, one is about stigma and smoking, and the other one is about stigma and weight.
In public health there has been a move within tobacco, and more recently within food, to try to stigmatize behaviors. So there is what’s called a tobacco denormalization approach – in essence the idea is to stigmatize smoking so it becomes socially unacceptable. At the population level this has seemed to be successful, but you still see smoking persisting among Californians who arguably are the most marginalized already in the state. I’m interested in understanding whether being someone who has already experienced marginalization, by virtue of maybe your sexual orientation, by virtue of your ethnicity, to what extent you might respond differently to public health messages of stigma.
It’s a little bit less clear whether or not the approaches to combat obesity in the US are purposefully stigmatizing in the same way that tobacco policies are, but there is some evidence… This is taking a step back to understand experiences of weight stigma, particularly among people who also may have experienced stigmatization in other aspects of their lives.
3. Stigma and hacking
When you talk about stigma it makes me think about what it’s been like talking to all these people who are doing user experience research while I’m working in public health. Some public health research seems like user experience research, but it has this concept of misuse. Instead of making certain products delightful, making them uncomfortable [4].
Judd: I think it’s a good analogy. You could argue that while it’s true in general that UX research is about making products better, really a lot of it is trying to optimize for whatever the user’s goal is – information seeking, or social support, or social connection, or just satisfaction with the product. In a lot of ways that’s the same thing that public health researchers are trying to do, but they’re using different techniques. They’re trying to optimize for health.
Optimizing for health sometimes involves discouraging certain behaviors. We do that in UX research too. We strategically point users in certain directions based on the things they’re interested in and what they hope to get out of their interactions. And the other thing we do which is similar in a lot of ways to public health is pay a lot of attention to the ways that people break our products. And when I say “break,” I mean sort of creatively appropriate. From what I understand, people in public health are also interested in the ways that people appropriate their environment, appropriate medical technologies, appropriate substances in different environments to serve their needs.
Sometimes those needs aren’t very healthy, but focusing on the ways that people, to use the Facebook terminology, hack their environment – hack their health, hack their nutrition, hack their addictions – I think there are a lot of parallels. We could go really go down that rat hole and maybe we shouldn’t. I already feel like I went too far by talking about hacking our addictions (laughs).
You were saying something today about how you felt like ethnographic work may be more accepted in technology compared to public health.
Tamar: Yeah it’s funny because I’m leading this workshop for students who are pursuing qualitative work for their dissertations, and one of them asked me the first day of the workshop whether I thought that qualitative research was stigmatized in public health. My first reaction was to say yes, in some ways, because when quantitative researchers are reviewing our proposals instead of our peers, and probably for a number of other reasons, we feel like our work is stigmatized. But then again there are really popular journals in public health, like Qualitative Health Research, for example. That’s a really highly regarded journal, and they have qualitative researchers who have been trying to popularize this approach for a long time, so it’s a conversation that’s been going on for a while.
4. Qualitative research as art, science and handmaiden
Tamar: Whether or not you talk about qualitative research as being science though, I wonder how most people in public health would perceive that
I’m curious if you would say people think of qualitative research as science.
Tamar: It’s interesting because, yeah, I think qualitative research is scientific. One of the things that we talked about a lot in our program at Maryland was the systematizing of qualitative research, and you and I have worked together long enough, I’m sure we’ve talked about this a number of times, but about how you make qualitative research rigorous and credible. Or I guess some people would want to say ‘valid.’ Qualitative researchers tend to want to use different terminology to talk about the sort of ‘validity’ of their work so that we’re not using quantitative terms to talk about qualitative research.
It’s important to think about how to design a really rigorous study, issues of triangulation, and how to make sure that you’re taking a holistic approach to the research. You’re looking for diversity of perspectives, and you’re thinking about your analysis in a rigorous and systematic way rather than a haphazard way – so that you’re really testing your assumptions about your data and your analysis. But I don’t think that all anthropologists think that the qualitative work they do is scientific.
It seems like some qualitative researchers take almost an art perspective.
Tamar: Humanities…
Judd: To me it’s partly different goals. I would call them both science. I would call them both research. I just think they have different goals. How do you choose to represent the people you’re working with, and what conclusions are you going to draw based on the work that you do.
The funny thing about quantitative researchers thinking that qualitative research isn’t scientific, is that so much of quantitative research is also arguably not scientific. It’s not theoretically informed or hypothesis-driven. It’s not rigorous. It has the illusion of rigor because it deals in numbers and statistics, even if it is based on a naïve interpretation of the results. As long as your p value is less than .05, you’re solid, right? There’s no other insight you need. You did good science!
I know that it is relatively easy to do qualitative research that pays much more attention to the details and methods of practice, that is much more theoretically informed, and that is much more deliberate and thoughtful about the conclusions than that kind of quantitative research. So it’s tragic when quantitative researchers say that kind of thing. It reflects a naïve and to me outdated bias [5]. I don’t know why we have to argue. Why can’t we just be like, look, you can do both of these kinds of research well, and in fact maybe you should be doing them together. But you can also do them both badly. It’s okay that they just have different purposes, different kinds of data, and different goals in mind.
That makes me think of something you quoted that alluded to the difference between quantitative and qualitative. It was this interesting point, do you remember?
Tamar: Oh, Maxwell. I can send it to you. And he makes this point exactly, that you shouldn’t be arguing over qualitative and quantitative research.
(And here is the quote)
…quantitative and qualitative researchers tend to ask different kinds of causal questions…quantitative researchers tend to be interested in whether and to what extent variance in x causes variance in y. Qualitative researchers, on the other hand, tend to ask how x plays a role in causing y, and what the process is that connects x and y (Maxwell 2012) [6].
Judd: One of the really successful strategies that I’ve found at Facebook is both quantitative and qualitative. We’ll do a large scale survey about a topic, and that survey will produce some descriptive statistics, and it will also produce open-ended comments, because Facebook users love to write about things that they like and don’t like. Then we can take the descriptive statistics and do a rigorous analysis of the narrative material. What’s been so interesting about this, is it’s been a really effective way to communicate about design. The statistics let product people and engineers know something about the generalizability of the findings and how common things are in general, but then they look to the richness of the quotes to learn about what to do about the problems we have. What is the nature of the problem, what are people thinking and feeling. So I’ve taken to including them both, interweaving them around different themes, and it seems to be very persuasive.
It’s weird because – maybe it’s just because everything in the Bay Area seems to be about health, or it feels that way to me – but I’ve thought of public health as being relatively more open to qualitative research. But maybe it’s not.
Tamar: I used to feel like I knew the answer to that question, but I feel like I don’t really know anymore. For example at this AAA session (that Tamar will be presenting at) [7], the session that we’re talking about is called “Are anthropologists the handmaiden of public health?” In a sense, it’s about this. It’s maybe not about the qualitative/quantitative divide, because a lot of anthropologists do quantitative work, and ethnography can be quantitative.
So it’s not so much that, but more the orientation that anthropologists have to their work, or that qualitative researchers have also to their work, isn’t necessarily seen to be highly regarded. But not only that, but are anthropologists having to ask questions that are defined by people in public health, and not being able to ask the critical questions that they normally would be able to ask.
Maybe it goes back more to this idea of what is science. Sometimes people think, okay that’s interesting that you found that and that may explain some of these quantitative findings, but is qualitative research producing valid knowledge in and of itself.
That totally reminds me of Implications for Design, the Dourish paper [8]. It’s a similar thing about being the handmaiden to systems development.
5. Big Data and Small Data
One topic that we emailed about before was the Big Data thing. Like you’re (Judd) doing work with Big Data. Is there any relationship between qualitative and quantitative, or is there such a thing as Big Data and Small Data, or…
Judd: Yeah. I would say two things come to mind. One thing, and Coye (Cheshire) [9] has really been a champion of this, is making sure that, okay, Big Data is great, but let’s not forget about Small Data. There is this idea that unless your data is huge then you don’t have anything good to say [10].
I should send you the link to this cool blog post that I saw from Harvard Business Review. The point it was making was pretty interesting. In Information Science there is this hierarchy, which is data, information, knowledge, wisdom. A point from this post was that Data Science is a hot field, everybody wants Big Data, but data is at the bottom of the pyramid. The place where you make use of it, where you have the insight and design, is at the top.
This is the part where I’m inserting myself. In between is the place where you have to understand people. In between is anthropology and sociology and psychology and economics and all those things. Data science and Big Data are useless without the translation that moves the data to the insight, and to do that you have to be an anthropologist or a sociologist. Arguably you have to mix methods and think about Small Data too.
(CC BY-SA RobOnKnowledge)
DIKW Pyramid
In between is the place where you have to understand people.
Cheerleader Pyramid
(CC BY-SA-NC A. Trumbly)
Tamar: It’s funny because when I think about the distinction in my world between Big Data and Small Data, Big Data is 350 interviews. Small Data is my pilot study. I’m not familiar with Big Data in the sense that you guys would be – not on the scale of Facebook right, that’s really big data. So I don’t know what I would say about the open-ended question, about Big Data versus Small Data.
Judd: Here’s a question for you. So part of the thing about Big Data is the idea that it requires the use of new tools and techniques to make sense of just terabytes and terabytes, right. But isn’t the same thing true of 350 interviews? I guess you could use the same analysis and techniques as you could on 20 interviews, but is there some different perspective you have to bring – because it’s not just 350. It’s 350 times all this rich data that came from all those interviews. So is there anything different about it?
Tamar: Maybe that’s the same argument about Big Data… big numbers aren’t necessarily always better.
Judd: I think that’s true in a lot of ways. I hear what you’re saying. I do think there are a lot of things that you can do with Big Data in the Facebook sense that you can’t do with smaller data. Or you can’t slice and dice in the same ways.
Tamar:You have a project, you’re funded to analyze the data on your project for three years. I mean you could write endless numbers of papers; you could create an entire career from 350 interviews, right? People are fascinating. Interview transcripts can be 100 pages long.
Judd: Maybe the real trick is to think differently, so not that you have to do either/or, but to think differently about how you use that data, what’s it for. What can you do with it? Because it is Big Data, it requires perhaps different techniques, just like in computer science we developed Hadoop and Hive and Pig to deal with the scale of the data, and apply new techniques to make sense of it and aggregate it in meaningful ways. Maybe you have the same problem with 350 interviews. Not that I have any solutions.
Tamar: Right, it’s different. In qualitative research, the human is the analyst. Jenna says this, right? She says something about that she only has so much ability to remember things when she’s going through her analysis [11].
Judd: ATLAS.ti as a technique for making sense of Big Data…
Tamar: Not exactly though, because one of the things that I always say when we do ATLAS.ti trainings is that ATLAS.ti, it’s not like SPSS. Not that I’m saying that the analyst is not worthwhile in a quantitative analysis using SPSS, but it’s different. You’re thinking about a theoretical idea, and you have a software program that allows you to aggregate all of these responses from people. ATLAS.ti is a data management tool. It houses your data so you can easily search, but it does not replace the researcher’s need to be in that data.
Judd: The computer scientist solution would be to count words and run it through the sentiment analyzer, you know. And that would be fine, but it would be very different. I feel like it might be naïve to think that there isn’t some solution which allows you to intelligently aggregate across a lot of interviews or find connections which are too difficult for the analyst to do him or herself because there’s just too much data.
Tamar: People need to be able to write proposals in a way to best do their qualitative research, and be able to say, you know what, for this particular research question, I need 40 interviews. I’m going to do a few more if I don’t hit theoretical saturation before that, or I’m going to do a few less if I hit theoretical saturation earlier.
The tradition of research that has historically been funded by NIH is fundamentally different than qualitative research. The reviewers for qualitative research proposals are increasingly becoming, I think, more qualitatively-oriented, but not always.
I keep reading Jenna’s posts, I love her series on Big Data [12]. There’s a paradox when I think about Small Data and Big Data. In some ways qualitative data is super huge. Maybe it looks small, like you need more data, but really you have so much data. Everything is data.
Tamar: Right.
Judd: How do you count? Do you count number of people you interviewed, do you count number of minutes of discussion, do you count number of words in transcripts? It can get overwhelming so quickly.
Tamar: And also you have tremendous amounts of narrative associated with a single case. For example, if I meet a person before I interview them, I write notes about the meeting. Then you do the interview, you write notes about the interview after you finish, and then you get the interview transcribed. Then oftentimes you’re memoing when you’re reviewing your transcripts, so then you have all sorts of memos associated with one transcript. It’s funny because when you guys first started talking about Big Data, I had the exact same thought. I know in your world Big Data means something, but in my world Big Data means something different.
Judd: I think that’s great. I don’t think it has that specific a meaning.
Tamar: You don’t think there’s an assumption that people make in your field about what it means?
Judd:Yeah, I mean I think generally people measure it by the megabyte. But if you think more abstractly, it’s data which is at the scale where you have to have different techniques to make sense of it.
____
[1] -Antin, J., & Cheshire, C. (2010). Readers Are Not Free-Riders: Reading as a Form of Participation on Wikipedia. Proceedings of the 2010 ACM Conference on Computer Supported Cooperative Work (pp. 127–130).
[2] -Antin, T. M. J., & Hunt, G. (2012). Food choice as a multidimensional experience. A qualitative study with young African American women. Appetite, 58(3), 856–863.
[3] – Antin, T. M. J. (2011, September). Contradictions in Food Choice and Body Image: Implications for Obesity Prevention. University of California, Berkeley.
[4] – More about fear and delight here.
[5] – Looking at you, Bob Garfield.
[6] – Maxwell, J. A. (2012). Qualitative Research Design: An Interactive Approach (Third ed.). Sage Publications, Inc.
[7] – “Have Medical Anthropologists Become the Handmaidens of Public Health?” at the American Anthropological Association conference in San Francisco on Saturday Nov. 17th.
[8] – Dourish, P. (2006). Implications for design. Proceedings of the SIGCHI conference on Human Factors in Computing Systems (pp. 541–550). Presented at CHI 2006.
[9] – Heather writes about Coye Cheshire’s work in A sociologist’s guide to trust and design.
[10] – Everything counts in large amounts (youtube).
[11] – Jenna writes about the limitations of memory in Small data people in a big data world.
[12] – Read Jenna’s complete series at The Ethnographers Complete Guide to Big Data, parts 1-3.
Thanks for your post, I included you in my weekly favourites this week. http://wp.me/p2Oz5B-22
Thanks!