Welcome to the Advanced Linear Models for Data Science Class 2: Statistical Linear Models. This class is an introduction to least squares from a linear algebraic and mathematical perspective. Before beginning the class make sure that you have the following: - A basic understanding of linear algebra and multivariate calculus. - A basic understanding of statistics and regression models. - At least a little familiarity with proof based mathematics. - Basic knowledge of the R programming language. After taking this course, students will have a firm foundation in a linear algebraic treatment of regression modeling. This will greatly augment applied data scientists' general understanding of regression models.
The singular normal distribution
Loading...
From the course by Johns Hopkins University
Advanced Linear Models for Data Science 2: Statistical Linear Models
24 оценки
Johns Hopkins University
24 оценки
From the lesson
The multivariate normal distribution
In this module, we build up the multivariate and singular normal distribution by starting with iid normals.
Meet the Instructors
Brian Caffo, PhDProfessor, Biostatistics
Bloomberg School of Public Health
So the multivariate normal distribution in fact just isn't rich enough for
the collection of distributions that we need even if we are going to assume that
their underlying outcome variables are normally distributed.
Let me give you a simple example of what I mean by this.
So take as an example, I have a vector x1, x2 which is,
both of those are scalars, is multivariate normal
with mean mu1, mu2 and variance matrix sigma.
So let me define A to be equal to 1,1,0 0,0,1.
Then matrix A times x1,
x2 works out to be x1,
x1, x2, x1, x1, x2.
So this can't be multivariate normal because the first two entries
are just the same one repeated twice.
And another reason it can't be normal
is the variance matrix which is A sigma
A transpose, is not full rank.
So it's actually not invertible.
So you couldn't even write out the normal distribution,
which remember requires the inverse of the variants in that exponent there.
So what's going on here?
The real problem here is that the matrix that I'm multiplying my
multi-varied normal vector by is not full row rank.
So when you multiply it by a matrix that's not full row rank
then you wind up with not a normally distributed random variable, you wind up
with what we're calling a singularly normal, a singular normal distribution.
The reason for calling it singular is, so singular normal.
The reason for calling it the singular normal is because
the variance matrix is singular, it's non-invertible.
And this is an important distribution for
us and I'll give you an example of when it's important.
So the standard assumption that we're going to make in regression is that our y
is normally distributed with mean equal to x beta and
variance equal to sigma squared I.
So take the residuals which are I minus H of x times y.
Look at this matrix right here, I minus H of x.
It's actually not full rank and
the reason I know that is because it's symmetric and idempotent.
And for symmetric idempotent matrices, the trace equals the rank.
So, if I take the trace of I minus H of x, that's the trace of I
minus the trace of x,
x transpose x, inverse x transpose which is n for the trace of I.
And then I could move this x over here and
then I get x transpose x inverse times x transpose.
So I get the trace of A p by p identity matrix which is
the trace of a p by p identity matrix is p.
So this matrix is m by n, but is of rank n- p.
So, we're going to wind up with the same problem.
Where we have a variance/covariance matrix that's not a full rank.
So the residuals, another way to see this, so the residuals,
even if my y is multivariate normally distributed,
my residuals are actually not multivariate normally distributed,
even though they are a linear combination of my vector y's.
And some of you might find this surprising because you might be,
already been doing regression a lot.
And had some practice of checking normality of your data
by checking normality, the apparent normality of your residuals.
And it's not like this is a bad practice, because when n is much larger than p,
your residual should be approximately normally distributed.
So it's not like this is a bad practice.
But as a matter of theoretical fact, your residuals are not,
are guaranteed to not be normally distributed.
Another way to see that the residuals can't be normally distributed is consider
the instance where we include an intercept.
If we include an intercept then the sum of our residuals is zero, however for
a normal vector, the sum of a normal vector has to be a normal scalar and
a normal scalar can't take a particular value with probability 1.
So it can't possibly be normal if it has that
kind of linear redundancy built into it.
And in fact there's p linear redundancies built into the residuals and
so there's many different ways you can create,
there's many different ways that you could create a vector,
a linear combination of these residuals that is a constant.
A full rank linear combination of the residuals that is a constant and
so you could see that that can't possibly be multivariate normal.
So the singular normal distribution was given
its name because of situations like this.
We needed something that included the normal as the special case,
the multivariate normal is a special case, but
then also encompassed all these other settings that we need.
So we define a singular normal as any linear transformation
of a multivariate standard normal.
And so in this case, that means any linear transformation of a non-standard
normal because we know that a multivariate
normal is a simple transformation of a standard normal.
So, take for example our case here.
We know that y is equal to x beta plus sigma times z,
where z is a multivariate standard normal.
So our residuals, our residuals e is equal to I minus H of x,
times x beta plus sigma z, where z is a standard,
of collection of IID standard normals.
So it satisfies the definition of being singular normal.
And the singular normal distribution carries over a lot of the properties of
the normal distribution that we would like.
For example, linear,
first of all any linear combination of singular normals is singular normal.
So it doesn't, full rank, or not full rank, it's singular normal.
The second thing is absence of covariants implies independence just
like in the multivariate normal, and all of the marginal and
conditional and subgroups of random variables
from the singular normal distribution, they're all also singular normal, okay?
So it carries over a lot of the properties that you'd like from the normal
distribution, but then takes away this property that you have to have full row
rank linear transformations in order to maintain the distribution.
It takes that property away at the expense of then we have random
variables that have linear redundancies in non-invertible covariance matrices.
Okay, so the singular normal distribution is an important distribution and
we’ll use it kind of frequently.
Ознакомьтесь с нашим каталогом
Присоединяйтесь бесплатно и получайте персонализированные рекомендации, обновления и предложения.
Coursera делает лучшее в мире образование доступным каждому, предлагая онлайн-курсы от ведущих университетов и организаций.
© Coursera Inc., 2018 Все права защищены.
