Online reputation: it’s contextual

This post is the first in a new category for Ethnography Matters called “A day in the life”. In it, I describe a day at a workshop on online reputation that I attended, reporting on presentations and conversations with folks from Reddit and Stack Overflow, highlighting four key features of successful online reputation systems that came out of their talks.

A screenshot from Reddit.com's sub-Redit, "SnackExchange" showing point system

We want to build a reputation system for our new SwiftRiver product at Ushahidi where members can vote on bits of relevant content related to a particular event. This meant that I was really excited about being able to spend the day yesterday at the start of a fascinating workshop on online reputation organised by a new non-profit organisation called Hypothesis. It seems that Hypothesis is attempting to build a layer on top of the Web that enables users, when encountering new information, to be able to immediately find the best thinking about that information. In the words of Hypothesis founder, Dan Whaley, “The idea is to develop a system that let’s us see quality insights and information” in order to “improve how we make decisions.” So, for example, when visiting the workshop web page, you might be able to see that people like me (if I “counted” on the reputation quality scale) have written something about that workshop or about very specific aspects of the workshop and be able to find out what they (and perhaps even I) think about it.

The organisers write that a reputation will be “a way for the user community to collectively calibrate the contributions of its members”. And if work of the new system will be “annotating” content on the web, then the reputation model will be an important part of that system. It turns out that calibrating contributions is not as easy as developing a scale and then marking a measure on a measuring jug. First you have to work out what the measure is. When is comes to peer production projects, the goal might be an vibrant volunteer community that comes together to produce something of public value. Wikipedia, for example, wants to see a growing volunteer community working together to build and improve a free encyclopedia, especially in areas that the encyclopedia is weak. Ushahidi, on the other hand, might want to see volunteers deploying and organizing around content in order to improve decision making and effective action in crisis situations.

When co-founder and general manager of the tremendously successful Stack Overflow and Reddit talked yesterday about how they developed their reputation systems, I was struck by the organic nature of their reputation model building process. Building reputation systems, it turns out, relies on an effective process more than a fancy algorithm. Successful codified reputation systems like those used by Stackoverflow and Reddit have developed their codes the way doctors grow skin on different parts of the body in order to use on other parts. Organically, along with the community, evolving in a process of increasingly shared responsibilities. Just the right amount of adherence to what the community currently values and how they already distribute rewards and attention, with just the right amount favoring or weighting of activities and values that achieved desired communal goals.

General Manager of Reddit, Erik Martin, spoke about the evolutionary process of the community. Reddit started purely as a link aggregator where users could vote stories up or down. After that came comments, then the ability for users to create their own topics. Then there was the ability for users to write their own posts, and the growth of users building new sites on top of the Reddit platform (there is even a radio station and the “University of Reddit” in this category!). Users have the ability to create “sub-Reddits” on categories that Atwood said “we never would have thought of”. Reddit has a karma system made up of up and down votes but when managers of the site enabled users to roll their own CSS, they saw users of sub-Reddits doing really interesting things to moderate discussions. In one sub-Reddit, moderators verify certain users as experts in specific fields and their comments are then highlighted. Members were able to innovate in this way specifically because of their ability to edit the CSS. Another interesting example is “SnackExchange” where Reddit users share snacks from around the world. When they have successfully exchanged a snack, an image of an AK47 appears next to their name to denote that they have fulfilled the promise to exchange snacks (I can’t wait to join and ask for rusks!).

Jeff Atwood, Co-founder of Stack Overflow talked about the focus on educating newcomers about the culture and practice of asking questions on the site. Stack Overflow is actually just one of many different sites from the company, Stack Exchange. Stack Exchange enables members to lobby for new content-specific sites in a place called ‘Area51’ where you can see the stages of new communities (vote for Libraries and Information Science!). Atwood says that the reputation model they developed weighted answers more heavily than questions (Stack Overflow enables people to ask questions about programming challenges) which hints at how codified reputation systems can work well to encourage particular types of behavior that will serve the overall goals of the community. Questions are easier to ask than to answer, for example, and if there were too many questions and not enough answers, the ecosystem would fail. Moderators are elected and have significant power to edit and remove things on the site. The platform uses both reputation scores and badging systems which Atwood (jokingly?) described as “meaningless rewards for doing things” that make everything you do on Stack Overflow “a lot more fun”. Reading the FAQ has a badge, for example. Filling out your user profile does too.

Atwood said that they’ve learned that moderators need a good mix of badges and reputation score to be good community facilitators and that reputation score can be a bad proxy for leadership qualities with users voting for members in community elections primarily because of their high scores. He also noted that a key principle behind their badge and reputation scoring system was that as you progressed through the site and through different experiences, you always had something interesting happening. He stressed that users are the ones who helped build the system and that they are essential to its functioning, even though they had to be educated about their role in the health of the community. “If you just make someone into Superman, they will start running into buildings… we didn’t teach them how to do good and fight evil (at first).” Orientation and education are really essential to that partnership.

Four principles stood out to me as I listened to Atwood, Martin and other reputation experts talking yesterday that might be useful in developing codified reputation systems.

1. Reputation systems must be tied to governance and community responsibility and control. The organisation needs to trust the system enough to say that growing reputation will equate to growing responsibility for governing and maintaining the system. This is the only way to keep members active and engaged in the long term.
2. The corollary of this is that the codification of the reputation system must be driven by the community. Users are not going to accept a system that doesn’t reflect the way that they use and understand the goals of the system.
3. Reputations systems are not static. They should be about rewarding a person’s work on the project over time, in addition or perhaps parallel to them taking on more leadership roles and administrative privileges.
4. Reputation systems are contextual. I heard this throughout the day with many of the participants talking about how different projects have wholly different ways of rewarding and recognizing different activities. Using one standard is difficult. It may actually be destructive, especially when it serves to surface content using only a particular type of lens.

Listening to the day’s proceedings, I realised that when people talk about Wikipedia not having a reputation system, they’re missing that Wikipedia does have a reputation system. It’s just not codified or transparent (which probably explains a lot of its problems). Every community has ways of determining who should get the most attention and what types of activities should be rewarded. A reputation model that codifies such a system is what we’re actually hearing about when we hear about new “reputation systems”. But by seeing reputation systems and codified reputation systems as the same thing, we start to see a community without a coded system like a person missing a leg. When you think of it this way, the solution is often seen as merely attaching a spare leg (spare legs are stored in multitudes in my imagination) and hoping that it will miraculously attain all the functionality it needs.

As Chris Dellarocas from Boston University said in his talk later in the day, the focus on different aspects of a reputation system rely on an understanding of what one wants to achieve with such a system. Communities might use reputation systems to achieve one (or more) of four goals:

1. Trust: encourage good and discourage bad behavior, and provide incentives for quality contributions
2. Filtering: assisting users in finding quality content
3. Matching: assisting users in finding content and members that are good for them (i.e. you might not find the content you’re looking for with a search but you might find people who might help you to find relevant content, especially when it is subjective area)
4. Participation and loyalty: give users reasons to join and stay in the community

Knowing what your goals are is really important to the design of the system, said Dellarocas. Also important is understanding what motivates users (he divides motivations into the three: love, glory, money – looking forward to that paper!)

By the end of the day, I was left with a very clear sense of how important ethnographic methods could be for a project like this. Being close to the experience of users, tracking and observing what motivates behavior, as well as truly understanding the quirks of certain communities are all the things that ethnographers do well. If you start without an understanding of the pecularities of what a community values and how it distributes attention, you’re going to end up with a new leg that may look good but doesn’t function very well. I was also pleasantly surprised by how many economists and computer scientists said how they had abandoned game theory and other abstracted ways of looking at these problems when they realised how un-generalizable reputation really is.

And the quote of day?

“Game theory imagines that humans are perfect information processors. They are not.”

Featured image by MervC licensed under a Creative Commons Attribution Non-Commercial license on Flickr

Translation

English中文(简体)FrançaisEspañol

by

You can help improve the automatic translation by checking off “Edit Translation” after you select a language. Thanks!