0:01

Hey guys.

Now we are going to talk about those mysterious beasts named

eigenvector and eigenvalues.

And just to make things easy so we don't have to write things out too much,

we're going to abbreviate eigenvector with ev with a little vector sign.

An eigenvalues as just ev.

So, eigenvectors, eigenvalues.

Okay, so what are these things?

What are these mysterious beasts.

Well, first of all, the eigenvalues and

eigenvectors are properties of a matrix.

Specifically, they're properties of a square matrix.

So, an n by n matrix.

And it will turn out that the eigenvalues

are invariant to a change of basis.

So, we saw in the change of basis video that you could pick

a new coordinate frame, a new basis and represent your matrix in that basis.

And different representations will give you different

groups of numbers in your matrix.

However, the eigenvalues of the matrix

are the same regardless of what representation your matrix is in.

1:29

So, that's going to be a very useful property later on.

Okay, but since we're taking about matrices,

let's remind ourselves of what a matrix is, or what it does.

So, what does a matrix do?

Let's say we have the matrix A, what does it do?

Well, a matrix takes in vectors, that's v1, and maps in to other vectors.

So, maybe when you multiply A by v1 you get this new vector, Av1.

2:12

So, what the matrix A is doing to each vector is

rotating it by a certain amount and then stretching or shrinking it.

So, for V1, it rotates at by maybe 30 degrees and

then doubles its length and something similar for V2.

Maybe it rotates it by 20 degrees and then scales its length by two and a half.

So, what does A do?

It rotates and stretches or shrinks vectors.

2:46

So, one way to talk about A is to make a big list of all the vectors you can

possibly think of and then say how much each is rotated and

each one stretches or shrinks when acted upon by A.

And so, you can do that if you have a lot of time on your hands, but it's really not

a very good idea because the list of all possible vectors is infinite.

Instead what we can do is to concentrate on a very specific set of

vectors or a very special set of vectors if you like.

And talking about how each one of those rotates or stretches

will actually completely describe what A can do to any arbitrary vector.

So, let's say we have our matrix A and

we're trying to figure out what the heck is going on with it.

And we know that if you have just some lame vector over here, v sublime.

When you multiply A times B sublime,

4:53

what it does is to shrink the vector.

So, maybe that's A times v2 special, so

all the special vectors, A just shrinks or stretches.

It does not rotate them, and so

what we have is that for a matrix A of dimension N x N.

So, remember we're dealing with square matrices now,

there are at least N vectors that

are only shrunken or stretched by A.

So, an A acts upon these vectors, it only stretches or shrinks them,

it does not rotate them and so these are special vectors and you know what?

In German, the work for special is eigen, or something like that.

So, these are called eigenvectors, and the specialness,

well they're special for a lot of reason.

But the first reason they're special is that A only stretches or shrinks them.

So, let's call the set of eigenvector.

E1 through en.

So, may be this special one was just equal to e1 and

this special two was equal to e2.

So, if the matrix A only stretches or shrinks eigenvectors,

we can write out A times an eigenvector.

Let's start with the first eigenvector, is equal to some constant times that vector.

So, the output is in the same direction as the input.

And we have A times e2 is equal to maybe a different constant.

Times that eigenvector.

6:57

So, in our case e1 would have a landa one grater than one, because it got stretched.

And e2 would have a landa two less than one, because it got shrunk.

And the landas are called eigenvalues.

The eigenvalue is the amoun that the eigenvector is stretched by or

shrunken by, when it's multiplied by A.

7:30

Okay, and now I'm not going to prove this to you but it's something you should

do for yourself at some point if you are into this math stuff.

But if A is symmetric, so the entries

below the diagonal are reflections of the entries above the diagonal.

That means that it's eigenvectors will be orthogonal.

So, if A is symmetric And we drew its igenvectors,

there will always be a right angle between them.

And that you weigh her in 3D space,

there would be a right angle between all of the eigenvectors.

All of the eigenvectors would be mutually orthogonal to one another.

So, what I claim is that in this case, knowing the eigenvectors and

the eigenvalues tells you what A will do to any vector.

So if we know all of the eigenvalues and the eigenvectors,

how can we figure out what A times an arbitrary vector will be.

Well, since our eigenvectors are all orthogonal and

we usually take, I forgot to mention this.

We usually normalize the eigenvectors so that they have length one.

9:01

Since they're all orthogonal to one another and there are at least n of them,

we can write v as a linear combination of them or a weighted sum.

So v might be v1 times e1 + v2 times e2 plus so

on and so forth plus vn times vn.

And plugging that into our

equation gives us Av = A(v1e1

+ ...+ VN, BN).

And we can use a distributive property to say that

that equals V1 times Ae1 + ...+VN times AeN,

where I've move to the Vs to the left of the As because they're just numbers.

So we can move them around without any problem.

But look at all these Ae1 and Ae2.

Since there are eigenvectors,

we can replace those with lambda e1 and

lambda e2 and so one and so forth.

To lambda N, sorry that's lambda 1e1, all the way up to lambda, NeN.

And of course keeping the v prefactor out front.

10:29

So, just like that, we have figured out what Av equals,

just by knowing the eigenvalues and the eigenvectors.

Okay, okay, so let's say that we have some symmetric matrix and

it has a couple of eigenvectors.

So e1 and e2.

And maybe, in this basis, e1 is let's say,

3 over square root of 10, 1 over square root of 10.

And in that case, e2 would be minus 1 over square root

of 10 3 over square root of 10.

And so that's an example of the actual numbers that you might put in your

description of your eigenvector.

11:34

And so the actual numbers that go into your eigenvectors

depend on what basis you're looking at.

However, in a certain way, even though the numbers changed,

there's still the same eigenvectors.

e1 is going to be right there no matter what my coordinate frame is and

e2 is still right there no matter what my coordinate frame is.

But most importantly, the action of A upon

those vectors is going to be the same regardless of what basis I'm in.

So as long a I write A in the correct basis, and

remember I can do that by saying A in the new basis

is equal to some change of basis matrix, times

A in the old basis, times the inverse of that matrix.

So as long as I write my A on the correct basis,

it's going to do the same thing to my eigenvectors.

And what is it do, well it'll scale them by their respective eigenvalues.

So this leads us to the conclusion that the eigenvalues,

so lambda 1 lambda 2 all the way to lambda n,

are independent of the basis you're in.

So the actual numbers you put down in the eigenvectors will depend on the basis, but

the amount that they are stretched or scaled by,

provided you represent A in the new basis, will not change.

13:30

And so, if you noticed, when we wrote out our eigenvectors

in terms of the actual numbers up here that was kind of messy,

in our standard x1, x2 basis.

I don't like square roots and everything like that and

I especially don't like negative numbers.

And I especially don't like it when every element of the vector has a weird

ugly number.

So the last question we'll ask is can we find a basis in which

the eigenvectors will be less messy?

And the answer is of course, we can.

14:37

So this is the eigenbasis.

And in this case, what is e1 in the eigenbasis?

Well that's easy, since it's only along one direction, the first direction,

it will just be 1,0.

And what's e2 in the eigenbasis?

Well, it's only along the other dimension, so that'll just be (0,1).

So this is a much cleaner representation of our eigenvectors than

representing them as stuff with square roots of 10 in them.

15:17

So how to think of this is as

the fact that the eigenvectors

of a matrix form the most natural

representation of the matrix.

And so why is it natural?

Well, what this means is that all of the special vectors,

so all of the vectors that just get scaled are those which lie a long

the axis you're using to represent your vector space.

So that's pretty neat, wouldn't you say?

And here is the icing on the cake.

We showed How to represent a matrix in a new basis.

And so if you're representing A eigen in a new basis,

that would be equal to x times the original a in your standard basis or

in your original basis, times x inverse.

So these are the.

16:58

And so how would we actually find this change of basis matrix?

Well, our new basis is just the Eigen vectors.

So we would have e1 transposed, so a row.

A row vector of our first Eigen vector as the first element.

And e2 transposed as the second row.

So, in this case, what do we say?

We said that E1 is equal to what?

3 / square root of 10, 1 / square root of 10, and

E2 was equal to -1 / square root of 10, 3 / square root of 10.

17:55

So let's quickly summarize what we've come up with so far.

First of all, the eigenvectors of a matrix

A are the vectors that are only scaled by A, and not rotated.

And we have that the scaling factors are called eigenvalues, so, ev's.

And we also had that knowing the eigenvectors and

the eigen values tells you how a acts on any

vector because that vector because that vector can

be written as a linear combination of the eigenvector.

And lastly, we had that the eigenvector's formed

the most natural basis for A because they caused.

A and eigen basis to be diagonal.

19:07

And lastly we had, this is very important, the eigenvalues

were invariant to a change of basis.

So you can pick any basis you want to represent a n.

And your eigenvalues will always be the same.

Because, remember, when we represent A in a new basis, what we're doing

is finding the transformation of A that preserves its action.

And its action can be defined by what it does to the eigenvectors.

Namely, multiplying them by their eigenvalues, I use.

So since the action of A is the same regardless of what basis it's in so

will the eigen values be.

So hopefully they're not too scary yet.

If you want to check if something is an eigen vector that's pretty easy we go.

So for example, let's say A 2, 3, 1 1.

And we want to check if the vector v = (1 1) is

an eigen vector of A, well what do we say?

So we can write

out Av = (2 1, 3 1) (1 1) and that's equal to, 5 2.

And 5 2 is clearly not a multiple of 1 1.

So v is not an eigenvector.

I'm not going to go through all of the details about how to

calculate eigenvectors.

These videos are more to explain the intuition behind these concepts.

But the starting point is to write out your eigenvector equation.

So A times e1 = lambda1 e1.

This could be rearranged such that A minus lambda times

the identity times e1 = 0.

And by solving this equation for lambda 1 and

e1 you get your eigenvector and your eigenvalue.

And in fact what you'll find is that this equation has several solutions.

Specifically, it will have at least as many solutions as the dimensionality of

your vector.

So if A is a 2x2 matrix and

it operates on two dimensional vectors you will have at least two eigenvectors.

So lastly just an example of why these are useful, well here are a few examples.

So here are some examples of when eigenvectors are useful So

one is it can help you decouple a set of differential equations.

Which is what they're used for in lecture two this week.

When we're finding the eigenvalues of the recurrent connection matrix.

It can also be used to decorrelate a random vector.

And so this is what we did when we were doing PCA.

In PCA what we had was a bunch

of random vectors drawn from some distribution.

22:24

And maybe these vectors were all correlated.

So here the x1 component is very collated with the x2 component.

When x1 goes up so does x2 when x1 goes down so does x2.

And it just so happen that Eigen vectors of the co-variance matrix

are these random vectors were arranged such that

if these vectors were from a gausion distribution they would suddenly be

decorrelated in the new bases, the eigen bases.

And it's nice so

to work with uncorrelated random variables when you're dealing with probabilities.

That's another instance when eigenvectors are useful.

23:06

Another example is in dynamical systems theory.

Specifically, in a linear system The eigenvectors tell

you the direction along which the motion of the system will be in a straight line.

And so that could be a very useful thing to know

when you're trying to figure out how a system behaves around a fixed point.

So that's all I've got for eigenvectors.

Again, the purpose of this was not to go through and

solve a bunch of complicated mathematical problems, but

rather just to give you a perspective of what eigenvectors were all about.