The future of designing autonomous systems will involve ethnographers
Note from the Editor, Tricia Wang: Next up in our Co-designing with machines edition is Madeleine Clare Elish, (@mcette), is an anthropologist and researcher at Data & Society, presents a case for why current cultural perceptions of the role of humans in automated systems need to be updated in order to protect against new forms of bias and worker harms. Read more about her research on military drones and machine intelligence at Slate. Madeleine also works as a researcher with the Intelligence & Autonomy Initiative at Data & Society which develops empirical and historical research in order to ground policy debates around the rise of machine intelligence.
“Why would an anthropologist study unmanned systems?” This is a question I am often asked by engineers and product managers at conferences. The presumption is that unmanned systems (a reigning term in the field, albeit unreflexively gendered) are just that, free of humans; why would someone who studies humans take this as their object of study? Of course, we, as ethnographers, know there are always humans to be found. Moreover, few if any current systems are truly “unmanned” or “autonomous.” [1] All require human planning, design and maintenance. Most involve the collaboration between human and machine, although the role of the human is often obscured. When we examine autonomous systems (or any of the other terms invoked in the related word cloud: unmanned, artificially intelligent, smart, robotic, etc) we must look not to the erasures of the human, but to the ways in which we, as humans, are newly implicated.
My dissertation research, as well as research conducted with the Intelligence and Autonomy Initiative at Data & Society, has examined precisely what gets obscured when we call something, “unmanned” or “autonomous.” I’ve been increasingly interested in the conditions and consequences for how human work and skill become differently valued in these kinds of highly automated and autonomous systems. In this post, Tricia has asked me to share some of the research I’ve been working on around the role of humans in autonomous systems and to work through some of the consequences for how we think about cooperation, responsibility and accountability.
Modern Times, 1936 [giphy]
The Driver or the System?
Let me start with a story: I was returning to New York from a robot law conference in Miami. I ordered a Lyft to take me to the Miami airport, selecting the address that first populated the destination field when I typed the phrase “airport Miami” into the Lyft app. The car arrived. I put my suitcase in the trunk. I think the driver and I exchanged hellos–or at the very least, a nod and a smile. We drove off, and I promptly fell asleep. (It had been a long week of conferencing!) I woke up as we were circling an exit off the highway, in a location that looked distinctly not like the entrance to a major airport. I asked if this was the right way to the airport. He shrugged, and I soon put together that he did not speak any English. I speak passable Spanish, and again asked if we were going to the right place. He responded that he thought so. Maybe it was a back way? We were indeed at the airport, but not on the commercial side. As he drove on, I looked nervously at the map on my phone.
Read More… The future of designing autonomous systems will involve ethnographers
Letter from the Editor: I am happy to announce the 
As computers become more human, we need to work even harder to resist the binary of computers versus humans. We have to recognize that humans and machines have always interacted as a symbiotic system. Since the dawn of our species, we’ve changed tools as much as 
Recent Comments