In general, peer review system, especially, like those, those on Amazon
online store, consist of three tops of rating, a text of review, and reviews of
review. Rating is simply just a number, usually
one to five stars. But it could also be one to ten.
It could be -three to +three. In any case, it is some kind of number,
usually integer and a review is a text. The text could be just a couple of lines
or could be five pages. And then reviews of review which could be
thumbs up and down which could be do you find it useful or not which is it again a
binary review. It could actually be a text of review on
this review. If you have a thread allowing rebuttal
back and forth. Okay, we can also implicitly understand
the quality of a review through for example, reputation score of the reviewer.
So there are many challenges in understanding how to best use a radio
system. One is the gate keeper problem.
Can anyone enter review or only those that actually can be verified if brought the
product or used the service. Is this anonymous, or is it ID based?
If it's anonymous, how can we prevent people from just entering random reviews,
or competitor, entering bad reviews, or the owner of the product or service
entering really good reviews? There's also the issue of scale.
If we give the choice of one to ten, often seven becomes the mode of the distribution
with the bi-modal distribution around it. From one to ten.
Somewhere around seven. If instead, the scale is one to five, the
star is different. Is that, however, large enough dynamic
range? If the scale is minus five to positive
five, it actually changes people's psychology, because minus sounds like
really bad. If the scale is only one to three, it
really narrows people's choices. So gate keeping and scaling really matters
before we can start even talking about radio systems.
Then, we face this challenge with mentioned number of reviews, this is a
particular focus point in this lecture. We know that more reviews is better,
assuming it is not random or, artificial reviews.
So, if it's truly, valid reviews, we want more.
The question is, how big is big enough for review population such that the average
rating is trustworthy. And in fact can we make the rating, the
average rating, modified, adjusted by the review population size?
Okay. And there's also the question of what is
the performance metric? This is very difficult.
First of all, depending on what you're reviewing.
Some of them are very subjective, like movies, in IMDB or Rotten Tomatoes.
Sometimes they are very objective, for example electronics on Amazon and
sometimes it's in between for example, hotels, or trip advisors, or restaurants
and open table. It's also driven by whose needs are we
thinking about? Okay.
Is it the reviewers? Or is it the owner or the seller of the
product and services? And how do we quantify the notion of
usefulness to different parties? The lack of universally agreed performance
matrix is one of the reasons why this problem is very challenging.
For example, you may disagree with root means square arrow L2 norm from last
lecture on NetFlix challenge. But most people agree this is a reasonable
proxy of the actual performance. Where as for Amazon average rating there's
no consensus yet. Now, given all these challenges, it might
seem that it is very difficult to come up with a good review system.
These average the entries by different reviewers is not going to work out.
However there are also well-documented anecdotal stories that show the potential
of reviews is more even as simple averaging.
A famous example is an experiment run by Galton in 1906 on a farm in Plymouth,
United Kingdom. So farm's festival and there were many
people, most of them were not farmers themselves and there was an ox.
That's my artistic rendering of the ox, okay?
On a stage, and there are many people standing next to the ox.
They each played a game of guessing how heavy the box is.
It turns out there are 787, who entered their guess on a piece of paper, and put
into a bin. And Galton say oh, it is impossible for
these people, who are not real experts on ox weight guessing, to be able to come up
with an accurate guess. So, he took all these numbers, into a
simple averaging, the number is 1,197 pounds.
It turns out that the actual weight is 1,198 pounds.
Almost exactly accurate. So this 99.9 percent accuracy is just
amazing. And this has been told and retold many
times as a story illustrating the potential of wisdom of crowds.
So, our question is, how can we generalize this observation?
And can this simple averaging carry over to Amazon?
