So now we actually use the, similarity values, before we do that, we have to
calculate the similarity values. And, when we actually do the similarities
we do them on the errors,um, that we take obtained in the baseline prediction.
We basically subtract out to figure out how off baseline prediction was, just for
the training set again, 'cause we can only use the training set.
And we augment baseline predictor with the similarity values that we obtained on
the errors. and so there's a couple reasons why do
that. First is that we really need to center
these values. At zero.
and the reason I've set it at zero is that we're kind of subtracting any, any
bias out already, by the fact that we're going to an error.
So the errors are going to include zero. And in a sense, they'll be centered
around zero. And, in order to do correlation, you need
things to be centered about zero. So then you can either have positive or
negative values, so if we did the money ratings themselves, which are from 1 to
5. You know we can never have negative
values, we'd only have positives, and that doesn't give us that sign
differential that we need. So we,we subtract down and we do them
from kind of around 0 to some Positives and negatives, and second reasons that
we're really, we're trying to correct for the errors here, right?
So this is an augmenting the baseline predictor with this, similarity.
the neighborhood method, really it's called.
And, so we, we want to correct for those errors, so we don't really want to do it
on the errors themselves because that's what we want to, sort of to go away.
And I mean we're not just going to add you know the errors back and give us zero
error on everything. We're going to do it in a way that makes
sense and is not reverse engineering. We'll see how we do that in a minute, but
first let's try to calculate some of the values.
So, here I'm showing the table the table that is.
So here I'm showing the table of error values.
And I've got this again, I got this by subtracting the predictions from the
actual values. So I took the actual values and I
subtracted the predictions. So if the predictions were higher than
the actual values, then this is going to be negative which means, well, we should
have made the prediction lower than we had it.
And if the predictions are less than the actual values, then this will be
positive, which means that the prediction should have been higher.
And so we can use the positive as a negative values accordingly.
but now for movies one and two let's just just apply that cosine similarity.
to do that we have to just figure out how the users that have rated both movies
because we can't use in the equation we can't have users that have only rated
one. And clearly we can't use those that have
rated none, none of the movies. but for instance we can't use A here,
because A has not rated movie two. we can use B, we're good because B has
rated both movies. we can't use C because, well this is part
of the, test set again. So we can't use the test set.
we can't use D, because this is part of the test set.
We can't E because we don't know it's value.
but we can use F because we know both of these.
So we have to use Only b and f for this. So, for movies one and two then, we're
just going to apply that equation that we had before.
We multiply b1 times f1. So we do -0.30 times -0.05, and then
remember we add. So we add the product of the terms, so we
do this times this plus this times this So then we add .17 times -0.58 and then,
remember we have to divide by the length, and then we divide by b, square root of
b1 squared plus b2 squared. So, this is negative 0.3 squared plus
0.17 squared. And then we multiply it by negative 0.05
squared, plus negative 0.58 squared. Then, if we do this out.
We get negative 0.0220 over. 0.3041 times 0.6044.
All right, so this whole top right here comes to this.
This comes to this. And this comes to this.
And then that Is equal to negative 0.11, okay.
So the cosine, or the similarity between movies one and two, is -0.11.
Remember now, we said that we have to see whether it's closer to -1, 0, or +1.
And if it's closer To negative 1 or plus 1, then it's useful, and otherwise it's
not. And so you see, this is really, kind of,
close to 0, it's somewhere around here. so that's, that's not very useful at all.
And so these movies, we would say are reallly not, very correlated.
Now lets try movies III and V if is another example.
So we will go through again and we will see III and V now a works so we'll just
do next it will say check mark we can use a because a has rated 3,, movie III and
this greater movie V. We can use B, again because B is rated
movie three, movie five. We can't use, three because this is part
of the test set. Sorry, it's not part of the test set,
this is, value that we don't know, we can't We can use, user d right here
because these we know both these values. We cannot use e because we don't know
this as part of the test set. And, we cannot use f because this is part
of the test set. But we do have three values now, 1, 2, 3
on each of these sides, so. we have to do a little more.
There's a little more terms here. So, now, we do this the same as last
time. Which is, we have 3 terms, that we're
summing instead of just two. So we do this times this, plus this times
this, plus this times this. So we have negative 1 times negative
0.43, plus. Negative, or plus, sorry, 0.25 times
negative 0.10 plus 0.25, again, times negative 0.10.
They actually turn out to be the same, for both of those.
then we divide by the square roots of this squared plus this squared plus this
squared times the square root of this squared plus this squared plus this
squared. So we do the square root of 1 squared
plus 0.25 squared plus 0.25 squared. Times the square root of .43 squared.
Notice that I'm omitting the square root, and that's because we, when you square
something it becomes positive again. So I don't need to write this garder
rits. + .1squared + 11 squared and If we do
this, if you do this whole thing out this entire multiplication I'll leave it to
you to actually run through the calculation but you get .79 right here.
And so that's closer to plus 1 right. So it's a positive correlation and it's
kind of closer to plus 1 so we would see if these movies are positively
correlated, right. So now we can use this then we can come
up with a full table of similarity values and here I'm tabulating this.
this is a similarity between one and two you can see.
the similarities between also 3 and 5, 0.79 we just found, and so on.
And now a couple things to note. First is that this table is symmetric,
right. So we've said before that things
sometimes aren't symmetric but here they are so the similarity from one to two is
the same as the similarity from two to one.
So that's why 1 and 2 and 2 and 1 are the same, just like you take 2 and 4, for
instance, and 4 and 2, same. So you can slap, flop it over this and,
if you mirror image it over it, it will be the same.
now in, in the next in the next segment, we're going to choose one neighbor or a
movie, right. So we're having the neighbor's movies
here, for each movie we're going to choose one neighbor.
and we could choose more, we could choose two, three or four.
But we've already made the math complicated enough.
But and that would get even more complicated.
So we'll just stick to choosing one neighbor for each movie.
Which will simplifly things a lot when actually go to do it out.
And so, basically down the columns right now we'll say movie one we want to try to
find the neighbor with the highest similarity.
Right, and so that's that would be three here.
Right, and these, a lot of the backgrounds now, of green, which are the
similarities, or the ones with the highest similarities.
So one would choose three as his neighbor.
And for that reason, and again where we were finding the magnitude so we want the
magnitude to be the highest. So even though this is negative, very
negative its still really negatively correlated which is a, a useful thing and
now for 2 we are going to choose 1, we choose between point 11, point 741 and
point 88 And therefore we're going to choose this.
Even though again it's negative it's still the highest in magnitude.
For three, three would choose one because it's got negative 0.82, which is higher
than any other values in magnitude. Four would choose two just like two chose
four. Those are actually perfectly.
negative correlation [UNKNOWN] one other. So, four and two.
And the reason that they're actually perfectly negative correlation is because
there's only one value that we're multiplying here.
And so that's not really that great. I mean, normally you need more data
before you say that two things are perfectly negative correlation.
But here that's just the way it turns out.
Now five will actually choose two to be his neighbor right.
This .88 is higher than the other values. And notice twodid not choose five it
chose four. but five choose two.
So they don't have to choose each other. even though the table is symmetric, they
don't necessarily choose each other. And we could define other metrics, for
instance, we could say, all right, well I'm not going to use similarity at all
unless the similarity value is higher, than like, 0.9.
For instance. And then in that case, we would only use
one neighbor for one pair here, we wouldn't use similarity at all.
Some people do taht because sometimes it makes sense to say well, unless
similarity is high enough, I'm not going to use it.
but here we're just going to choose the most similar and just use that to do our
calculation.
