My Friend’s Boss Has Little Relevance To Me

posted by eric

trustmail.jpgSort by trust in TrustMail.In this paper Jennifer Golbeck and James Hendler outlines a “reputation network analysis” system for email filtering. The concept is simple: rate everyone you know on a scale from one to ten, and your email application will calculate a “reputation score” for each and every incoming mail message (using a local trust metric algorithm and some FOAF-magic).

Wouldn’t it be great if you could sort your inbox by “relevance”? I thought so too, but then I took a look at my own inbox while trying to think of a rating for each mail that was there. And guess what–it turned out to be quite hard. In fact, in some cases I couldn’t come up with a rating at all, and in other cases I could think of several different ratings for the same person.

Why? Well, we know already that reputation demands context. So whenever I thought of a rating, I had to artificially place the person in a context, which felt somewhat awkward, arbitrary and at times plainly wrong. Which context to choose? What if there were multiple? What if there was none at all?

This is also the reason why the system probably wouldn’t work well anyway. Reputation scores that are inferred by the algorithm are calculated without taking context into account. So let’s say my friend’s boss sends me a mail. I trust my friend, my friend trusts his boss. And yet, the mail from my friend’s boss has little relevance to me. Both reputation scores are accurate in themselves, but when we collapse the contexts around them, they loose their meanings.

So, we need a way of saying “I trust you in this particular context”. But then we run in to issues of etiquette and fingerspitzgefühl. By saying explicitly “I trust you in the context of work” I’m tacitly saying “I don’t trust you outside of work”. Apart from the high cognitive load, I’m actually being rather tactless.

Needless to say, given that I couldn’t even trust my own ratings of my own friends, I’m a bit skeptical of the Trustmail approach. The same critique is valid for many other reputation systems. Clay Shirky puts it rather harsh but well: “Almost all the work being done on reputation systems today is either trivial or useless or both, because reputations aren’t linearizable, and they’re not portable. […] The world’s best reputation management system is right here, in the brain”.

August 21st, 2006

eBay Reputation Squabble Leads to Lawsuits

posted by eric

A $2.33 transaction followed by (incorrect) negative feedback resulted in a yearlong dispute. Gives an impression on how important reputation is to sellers on eBay. [via Opinity]

August 10th, 2006

Reputation and Fingerspitzgefühl

posted by eric

Reading texts like “Manifesto for Reputation Soceity”, “Toward a Private Digital Economy” and re-reading the whuffie parts in “Down And Out in the Magic Kingdom” has made me think again about reputation. So many questions! Where to begin? Are we moving towards a sort of hyperreal version of Tönnies Gesellschaft, or a cyberspacified Weberian Iron Cage? Can we position all people on the one-and-only Great Reputation Graph, or are there in fact several over-layering, intersecting, and incommensurable systems of reputation? Is reputation portable? How to measure Halo?

We talked with Auren Hoffman (CEO, Rapleaf) about their rating service the other day. They want to break out the buyer-seller-rating component of eBay–a bold act that is not without problems (Oh no, the silos fight back!). After all, one of the most important preconditions for building reputation is context. Like Tom Dell’Aringa puts it in a comment: “I can honestly tell you that a Rapleaf score would mean zero to me in eBay. I don’t care about what some 3rd party rep system states about a person on eBay. I want to know what their rep is inside eBay!”

I have a hard time figuring out whether RapLeaf’s very Web2.0ish service (api:s, openness, componentization, you got it) is the right way to go in this case. I believe dealing with reputation takes some fingerspitzgefühl. This is a tip from the RapLeaf blog:

“Going on a date? Worried you’re on the Dontdatehimgirl website? Offset that! Send her an email with your RapLeaf badge in the signature. She’ll see that you’re a good guy.”

Wow, isn’t that like putting your credit rating in a love letter? Or am I getting something wrong? It reminds me of the story about the woman who wants to go partying with her friends. Her husband says: “I trust that you won’t cheat on me”. This story illustrates very well what happens when talk about trust gets to explicit. In this case the woman gets seriously offended–saying something like that is a clear sign of distrust. [My supervisor told me the story. I believe he found it in Lagerspetz]

A Colombian artist told me the flipside of this story yesterday. Some time ago she walked into a store–in a small village where she’d never been before–to buy some milk. When she wanted to pay she realized she didn’t have any money on her. The owner of the store immediately offered her credit. “How can you offer me credit when you don’t know me?” was the obvious question to ask. “I’d rather lose my money than my trust” was the owner’s swift reply.

A village, a marriage, a date, an eBay auction. Context seems important. Fingerspitzgefühl too.

[UPDATE: Noted that the part on the date quoted above has been removed from the blog post over at RapLeaf. Good move.]

The System of Reputation

posted by alex

Later today we are meeting up with RapLeaf CEO Auren Hoffman so here are some current pre-talk thoughts. When talking about reputation systems I think it is important to keep two things that aren’t mentioned that often in mind:

1) Input is unevenly distributed
2) Reputation need not be explicit

According to this talk there has been much research concerned with the functionality of reputation systems such as the ones on eBay or Amazon. Many studies show that the users inclinations to provide feedback is highly asymmetrical depending on if their experience is positive or negative. Users will give positive feedback on a deal gone right but will avoid giving negative feedback on deals gone bad. Why? Fear of retaliation.

Online reputation has become so important to people that the fear of upsetting your personal reputation stock value is simply higher than the drive to perform good for the community. Say that you just scammed me on Amazon and I just lost some money. The question is what I now have to gain by giving negative feedback about you? I have already lost my money and by giving you negative feedback, by means of reciprocation I am also gaining risk of your negative feedback retaliation and thus a decrease of my personal reputation score.

Conversely, if you actually didn’t scam me and the transaction went well, I might be inclined to give you a higher rating than necessary since you might reciprocate this by in return giving me a higher score (i.e. increasing my personal reputation stock value). So, we have a situation where we lack negative feedback and incentivize the polarization of positive feedback which naturally begs the question: Are reputation systems really useful at all?

Of course they are, but only to a certain extent. Moreover, one needs to take these system distortions into account when assessing the reputation value of people. On Amazon for instance, it seems to me that nobody that has a rating has one that is lower than say 90%. In other words, the 0-89 of the 100 unit scale are never used. In this scenario 91 is a terrible rating while 95 is OK and 99 is pretty good. Furthermore the rating can be seen as a proof of previous transactions that in itself provides a certain limited amount of trustworthiness.

Alex99reputationThe second issue I’d like to address is that in offline-life reputation is rarely explicated in the form of a numerical score. Yet, since the beginning of the net we’ve always had ways of hinting about a persons reputation anyway. Like I mentioned here, online, just like offline, we have certain social cues for reading people and assessing their character. In a forum setting most people gain reputation by the amount of posts they have made, how long they have been members, etc. In the days of social networking we can use connections to others to establish if a person is trustworthy or not. These cues aren’t explicit reputation cues or measurements but we still use them in this way–just like in real life. A numerical specific reputation value can be a quick and easy way to judge somebody but also, for the same reason, a highly motivating attribute to try to “game”.

Reputation will always be present, whether we have an explicit system for it or not. The question is–do we want it to be explicit and numerical? In the Amazon system, the inherent problems become a way of obscuring the explicitness of the system and translating it into an implicit cue about reputation. In other words, we are putting numbers on stuff to explain tricky phenomena and at the same time figuring out new ways of interpreting what these numbers really mean.

What Is Trust?

posted by eric

The deeper we dive into this study, the more we realize how vague the term “trust” really is. It becomes especially evident as soon as one starts to study a particular trust-related phenomenon (such as buyer-seller trust in online marketplaces). One quickly realizes that “trust” is often a much too general term to use–it is commonly used as a “catch-all” for more specific concepts. There’s clearly a problem of “semantic discrepancies” in the current discourse (but then again, that seems to be the issue in almost any hot enough discourse).

Trust, at least seen pragmatically, can be broken down into more precise concepts such as reputation, credibility, predictability, and consistency. The credibility people at Stanford have done a fairly good job of distinguishing credibility from trust, whereas Seligman tries to separate out “simple” predictability from trust. Many more similar efforts can be found in the literature. We’ll dig more deeply into these semantic issues within the next few days.

In the meantime, here’s an independent study commissioned by Rapleaf on online marketplaces that concludes that “posted ratings are the most important factor in determining their level of trust in sellers” (my emphasis). We hope to do an interview with the Rapleaf guys soon. Their plan is to make an eBay-like reputation system available through open API:s, in essence creating a global, open reputation system. I really think it’s about time, but it remains to be seen how well they can solve the obvious fraud problems

UPDATE: Here are the full details of the study.