So now we need to leverage these similarity values, and in order to do
that we need to use three tables that we found previously.
This right here, this table is just the original ratings that I'm showing.
But in addition we need the baseline predictor, we need the baseline errors
which again is by taking the subtracting the predictions out, and then we need the
cosine similarities, and which we just found.
So when we have these three things, we can actually leverage these similarity
failures. And so we're going to build what's called
the neighborhood predictor now, and it's, it's very simple once we have these three
tables to do the neighborhood prediction. We just have to look, look up a few
values in the tables. And so let's, let's just look at an
example first, and then we'll write out the general case of it and see how you go
to apply it in a general sense. Alright so lets lets take C five.
Right so we'll take, user C, movie five. So we wont be using this table.
We'll just, reference it if we need to, to see.
And so, with C five, we know we in the baseline predictor we had 2.60.
And, we know the prediction was too low by, 3, right.
So, we have a 2.60 as the baseline prediction for that, okay.
Now we need to do something with, the neighbor, right, because that's the whole
idea here, we want to use the neighbor. So since they're movie neighbors, we'll
try to look up movie five and we'll find movie five's neighbor.
So, movie five's closest neighbor is two. As we've said, cause this is the
neighbor, so we, we want to come down and find two, right here.
So, now, if we look in in two, 'kay the next question is did user C rate movie
two? And the answer is yes he did, and it's
not even part of the test set, because if it was, then we couldn't use the value.
Since it's is also part of the training data, it's still useful.
'Kay, so, user C did rate movie two. You can see right here the rating was
1.33. And that 1.33 was lower than this two,
right? Which is why this value is positive right
here. So it's indicating to us that we needed
to increase the prediction to get it to where I need it to be.
So now, what we want to do is we want to combine this, this prediction right here
for C five with the error that we got for C two, okay, because two is the neighbor
of five, and we need to, we're going to use two because we can't use the value
directly or else that would just be reverse engineering.
We have to use other values that are in the training set on the table.
Okay, because if we did, if we just used this value, right, if we just added C
five back and got zero prediction, then that wouldn't be very helpful for, for
instance, when we were using the actual values and we needed to find which are
either the test set or the the ones that we don't have because then there would be
nothing to add, right? If we were just going to add the values
back we wouldn't know what to add. And that's just the idea again that we
can't reverse engineer the test site. We just have to use other structure
within the table. So we find the neighbor, the nearest
neighbor is two, and then we find two, and we see that user C has rated movie
two. Okay, so this is and it's 0.67, is that
value, okay. And now we're going to add 0.67 to that,
and that gives us 3.27, 'kay. So now, sometimes we don't want to add
values. Okay, so we're adding right here this
um,neighborhood prediction value. Sometimes we want to subtract those.
And I bet you can probably guess when we want to add and when we want to subtract.
It dependes upon whether we have a positive or a negative correlation.
So really we don't care about the absolute values that are in this table.
We would if we were doing two, or three, or four neighbors, but since we're just
doing one neighbor, all we care about is whether the values are positive or
negative. So you could really just replace this
with either a minus or a plus, in all these values here and it wouldn't make a
difference to us, for these purposes. Okay, so now, let's consider another
example. Let's try user B and movie four.
So with user B and movie four, we have a baseline value of 3.50 and that's in the
test set. So, now this is, this is a case where we
couldn't actually add the error, because we don't know what the error is.
So we have 3.50, okay, now we look at four, four's neighbor is two, so four
also uses two as it's nearest neighbor. So, the question is now, has B rated
movie two? And the answer is yes, B has rated movie
two and it is part of the training set. Okay, so, we can use this value.
And this value is 0.17. So, what that's indicating to us is that
the predictor was too low by 0.17 for this rating of b for movie two.
Now here's the kicker. Two and four have a negative correlation.
'Kay, so since the predictor was too low for movie two, we're going to think that
it was probably too high for movie four. So rather than adding this value of 0.17,
we're going to subtract the 0.17 out. Okay, so we're going to subtract the 0.17
and say, you know, since this was indicating it was too low, it was
probably too high for number four, okay. And statistically speaking, we would say
that, probabilistically it is, probab, the probability is great that it's going
to be higher, that's the idea here. so then we subtract out 0.17 and we get
3.33. So this is for C five, this is for B
four. So now we can write out just a simple,
general equation, here, right here, is that the neighborhood predictor is going
to be the baseline plus or minus the error for the neighbor, and we, we add if
it's a positive correlation then we subtract if it's a negative correlation.
That's it. That's, so that's all we need.
We just need these three tables and then we can do all these calculations.
So we can try another example really quick, because I just want to illustrate
the fact that here with C, with C five for instance, we saw, we know that the
value is 3 over here. So we're going from 2.60 to 3.27 which is
overshooting the value, but it's still closer, right.
3.27 is closer to 3 than 2.60 is. And the same thing with B four, if we
look at B and four, the real value's 3, B four is 3.50 before, and now we're
subtracting and getting 3.33. So we are getting closer.
But, in general, we're not always going to get closer, and we just hope
that overall the RMSC gets lower. So sometimes, it's going to be a mistake,
it's going to go in the wrong direction, for instance.
So, as an example of that, let's take user F's rating will be 5.
So, this, this is 0.4 right here and 3.85s already pretty close to 0.4.
but for F five we said we know we would have 3.85.
And then if we look at five's nearest neighbor again, we said was two.
So, then we go into the table for two. F has rated two, and they're a positive
correlation so, we would actually end up subtracting.
And we add the error here, which is a negative value which means that we
subtract here, is saying the predictor was too high, then we get something like
3.27, and 3.27 is clearly in the wrong direction.
3.85 was closer so this is getting farther away.
And so it's not always going to give us better but there is nothing we can do
about that. We just hope that it overall gets better.
And the final point is that if if we get a time where the user, the neighbor has
not rated the movie, then we just don't use it.
Like we know for instance, that, five's neighbor is two, we'll take that example
again. So if we were doing, trying to do for D
five, right, and we have this value here, we could not use D two right, because
even though two is the neighbor, so we want to see what D rated movie two in
order to augment the prediction for five, but since that's part of the test set we
couldn't use that. So we would just keep the baseline
predictor the way it was instead of trying to add anything on at all.
