Before we talk about overdetermined systems where we have too many equations, we have to introduce the concept of orthogonality and inner product. So, the question is when are two vectors mutually orthogonal? The answer is simply when their inner product is zero, and this lecture is to show that these two are connected. Recall that the length of a column vector can be written as the square root of the sum of the squares of the coordinates. So, if we consider each point is in n space as the tip of a vector leading from the origin to that point, we can say that the length of the vector x is simply the distance from the origin, and this can be written as the square root of the sum of the squares or is the inner product of x with itself where x is a column vector. So, when we write x transpose x, here is x here is x transpose. If we multiply these together using the usual rules of matrix-matrix multiplication, we get the sum of the squares. The distance between two points is the distance between the vector representing those two points which would be the distance between those two tips of those two points. So, the distance from x to y, and we'll take it squared because we want to avoid square root symbol, is equal to the length of the vector from x to y or actually from y to x. Remember, the length of a vector is the inner product of the vector with itself so we can write it this way. Now, we can expand it using all the rules that we were talking about in our previous lecture, the distributed rule and various other rules is as x transpose x plus y transpose y minus 2x transpose y, and this is represented in this picture. So, the vector from y to x, you started off at this point, you go down until the origin, and then you go up to x, and this gives you the x minus y vector, and the distance between those two vectors is the length of this vector. We can look at minus y instead of starting at the origin. Instead of following y, we go in the minus y direction. If we start from minus y go to x, we get the vector x plus y. This is the negative of minus y so we follow y and then we follow x, so this is the sum of the two vectors. We notice that the x and y are going to be perpendicular to each other exactly when these two distances are the same, x plus y and x minus y. So, we write the equation x distance x to y is equal to the distance x to minus y. If we expand this just like before, we get the equation that x transpose x plus y transpose y minus 2x transpose y is equal to the same thing with plus 2x transpose y. We see that in order to be the same, these two items in the red boxes have to be zero. So, we conclude that x is perpendicular to y or orthogonal to y exactly when their inner product is zero, and the inner product is simply the sum of the element-wise products of the elements. So, if x and y are both column vectors, you can write this as x transpose y.
