My Friend’s Boss Has Little Relevance To Me
Sort by trust in TrustMail.In this paper Jennifer Golbeck and James Hendler outlines a “reputation network analysis” system for email filtering. The concept is simple: rate everyone you know on a scale from one to ten, and your email application will calculate a “reputation score” for each and every incoming mail message (using a local trust metric algorithm and some FOAF-magic).
Wouldn’t it be great if you could sort your inbox by “relevance”? I thought so too, but then I took a look at my own inbox while trying to think of a rating for each mail that was there. And guess what–it turned out to be quite hard. In fact, in some cases I couldn’t come up with a rating at all, and in other cases I could think of several different ratings for the same person.
Why? Well, we know already that reputation demands context. So whenever I thought of a rating, I had to artificially place the person in a context, which felt somewhat awkward, arbitrary and at times plainly wrong. Which context to choose? What if there were multiple? What if there was none at all?
This is also the reason why the system probably wouldn’t work well anyway. Reputation scores that are inferred by the algorithm are calculated without taking context into account. So let’s say my friend’s boss sends me a mail. I trust my friend, my friend trusts his boss. And yet, the mail from my friend’s boss has little relevance to me. Both reputation scores are accurate in themselves, but when we collapse the contexts around them, they loose their meanings.
So, we need a way of saying “I trust you in this particular context”. But then we run in to issues of etiquette and fingerspitzgefühl. By saying explicitly “I trust you in the context of work” I’m tacitly saying “I don’t trust you outside of work”. Apart from the high cognitive load, I’m actually being rather tactless.
Needless to say, given that I couldn’t even trust my own ratings of my own friends, I’m a bit skeptical of the Trustmail approach. The same critique is valid for many other reputation systems. Clay Shirky puts it rather harsh but well: “Almost all the work being done on reputation systems today is either trivial or useless or both, because reputations aren’t linearizable, and they’re not portable. […] The world’s best reputation management system is right here, in the brain”.
The idea of context for trust is one we are acutely aware of. If it’s not mentioned in the old paper you cite (2003, I think – I’ve done a lot of work since then), you can see long discussions of it in my newer papers. Of course it is necessary, but there is a trade off when building systems. People aren’t going to give a set of a dozen or more ratings to each person they know…it’s hard to get them to give ONE rating to people they know. So if you’re building something like TrustMail as a proof of concept, you go with what it’s easy to get, and that is single values.
The core idea is using trust for ordering items. It’s one that I applied to sorting movie reviews in FilmTrust, a project I did two years ago for my dissertation. The reviews from the most trusted people appeared first, and less trusted reviewers’ comments were later. Our studies showed both that people liked this (i.e. the ordering reflected what they thought was the correct order), and that the trusted reviews tended to correlate with the user’s opinion of the film better than the untrusted ones. The use of trust as an ordering mechanism was the point of the TrustMail paper as well, and it does work.
Shirky likes to criticize, but the fact is that a lot of these approaches *work*, even if they are not perfect solutions. Obviously the brain is going to be better than a computer. But your ability to compute trust / reputation stops with people you know. These social network-based approaches extend that knowledge to give you insight into people you don’t know. Sure, your friend’s boss may not mean much to you, but the information you’re getting here simply says “Someone you trust trusts this person highly”. I would argue that is better than knowing nothing about the boss. TrustMail is basically a small hack to illustrate how that could work.
However, I and others have done work showing that trust can be used in recommender systems to create better recommendations than some of the traditional means. Some of the work I cited in the TrustMail paper (Boykin & Roychowdhury) show that social networks used as spam filters can give dramatic jumps in the accuracy of filtering. Trust is an excellent tool for sorting, aggregating and filtering. You say it needs context and I agree. That is really trivial. Add context or, better yet, use it in a context-specific application where that is built in. There is a large and growing body of scientific research that shows the trust approach works.
I think trust is the key aspect in the discussion about the Wisdom of the crowds. I’m working on a social shopping application for wine ( http://www.adegga.com )and we’ve been dealing with some of this issues.
I’ve written about ( http://blog.delaranja.com/?p=637 ) how on social shopping you better trust your friends first (those you already trust) than the wisdom of the Crowds.
Jen Golbeck’s comment shows that in some cases that’s the way users expect it to be. On the other hand it’s good to know that users value having access to all the information (reviews) just ordered by level of trust.
I was just browsing for relevant blog posts for my project research and I happened to stumble upon yours. Thanks for the useful information!