Latent Variable Models

Loading...

From the course by National Research University Higher School of Economics

Bayesian Methods for Machine Learning

142 оценки

National Research University Higher School of Economics

Bayesian Methods for Machine Learning

142 оценки

Course 3 of 7 in the Specialization Advanced Machine Learning

Bayesian methods are used in lots of fields: from game development to drug discovery. They give superpowers to many machine learning algorithms: handling missing data, extracting much more information from small datasets. Bayesian methods also allow us to estimate uncertainty in predictions, which is a really desirable feature for fields like medicine. When Bayesian methods are applied to deep learning, it turns out that they allow you to compress your models 100 folds, and automatically tune hyperparametrs, saving your time and money. In six weeks we will discuss the basics of Bayesian methods: from how to define a probabilistic model to how to make predictions from it. We will see how one can fully automate this workflow and how to speed it up using some advanced techniques. We will also see applications of Bayesian methods to deep learning and how to generate new images with it. We will see how new drugs that cure severe diseases be found with Bayesian methods.

From the lesson

Expectation-Maximization algorithm

This week we will about the central topic in probabilistic modeling: the Latent Variable Models and how to train them, namely the Expectation Maximization algorithm. We will see models for clustering and dimensionality reduction where Expectation Maximization algorithm can be applied as is. In the following weeks, we will spend weeks 3, 4, and 5 discussing numerous extensions to this algorithm to make it work for more complicated models and scale to large datasets.

Latent Variable Models11:32

Probabilistic clustering6:32

Gaussian Mixture Model10:13

Training GMM10:41

Example of GMM training10:35

Meet the Instructors

Daniil Polykovskiy
Researcher
HSE Faculty of Computer Science
Alexander Novikov
Researcher
HSE Faculty of Computer Science

Welcome to week two of practical Bayesian methods.

I'm Alexander Novikov and this week we're going to cover the Latent Variable Models.

So what is the latent variables?

Why do we need them?

And how to apply them to real problems?

And the second topic for this week,

is the Expectation Maximization algorithm,

which is the key topic of

our course and this is a method to train latent variable models.

We will see numerous extensions of

this expectation maximization algorithm in the following weeks.

So, let's get started with the latent variable models.

Latent variable is just a random variable

which is unobservable to you nor in training nor in test phase.

So, latent is just hidden from Latin and as an example of those you can think.

So for example, some phenomenons like heights or lengths

or maybe speed can measure directly and some others can not.

For example, incidences or altruism.

You can't just measure altruism with some quantitative scale.

And so these variables are usually called latent.

And to motivate why do we need to introduce this concept into probabilistic modeling.

Let's consider the following example,

see you have an IT company and you want to hire an employee,

so you have a bunch of candidates,

and for each candidate, you have some data on them.

So for example, for all of them you have their average high school grades,

for some of them you have their university grades

and maybe some of them took IQ tests and stuff like that.

And you also conducted a phone screening interview.

So, your HR manager called each of them and ask them a bunch of simple questions

to make sure that they understand what your company is about.

Now, you want to bring these people onsite to make an actual technical interview,

but the problem is that you have too many candidates.

You can't invite all of them because it's expensive.

You have to pay for their flights,

for their hotels and stuff like that.

So, natural idea arises,

let's predict the onsite interview performance for each of

them and bring only those who are predicted to be good enough,

so how to predict to be a good fit for our company.

Well, if you're in the business for a while,

you may have some historical data.

So, for a bunch of other people,

you can know their features like their grades and their IQ scores,

and you know their onsite performs because you have already conducted these interviews.

Now, you have a standard regression problem.

You have a training data set of circle data,

and for new people you want to predict their onsite performance,

and you want to bring on their onsite interviews only

those whose predicted onsite performance is good.

However, there are two main problems why we can't apply

here the standard regression methods from machine learning.

So first of all, we have missing values.

For example, we don't know university grades for all of

them because Jack didn't attend university.

And it doesn't mean that he is not a good fit for your company.

Maybe he is but he just never bothered to attend one.

So we didn't want to ignore Jack but we want to

anyway predict for him some meaningful onsite and field performance score.

And the second reason why we don't want to use

some standard regression methods like linear regression or neural networks,

is that we may want to quantify uncertainty in our predictions.

So, imagine that for some people,

we may predict that their performance is really good and we

certainly want to bring them onsite and maybe even want to hire them right away.

But for others, the predict performance is not good.

And for someone, the predict performance can be for example of 50,

which may mean that this person is not a good fit for your company.

But it may also mean that we're just not sure about him.

So, we don't know anything about him and you asked the algorithm to predict

his performance and he returned you some number but it doesn't mean anything.

So in this case,

we may want to quantify the uncertainty of the algorithm in the predictions.

So, if the algorithm is quite sure that this person

will perform at a level of 50 out of 100 for example,

then we may not want to bring him onsite.

On the other hand, if some other guy predicted performance is also

50 but we're really uncertain about his performance,

then we may want to bring him anyway and see maybe we just don't know anything about him.

So, he may be good after all.

And the reason for this uncertainty may be for example that,

he has lots of missing values or maybe his data is a little bit contradictory or

maybe our algorithms just aren't used to see people like that.

So these two reasons,

having missing values and wanting to quantify uncertainty,

bring us to the needs of for probabilistic modelling of the data.

And as we discussed in week one,

one of the usual way to build probabilistic model is to start

with drawing some random variables

and then understanding what are the connections between these random variables.

So, which random variables correlate with each other in some way?

And in this particular case,

it looks like everything is connected to everything.

Like if a person's university grades are high,

it directly influences our beliefs about

his high school grades or his IQ score and this is true for any pair of variables here.

And the station where we have all possible edges,

like everything is connected to everything,

means that we've failed to capture the structure of our probabilistic model.

So we end up with the most flexible and

the least structured model that we can possibly have.

And in this situation,

we have to assume.

So to build up a probabilistic model of our data now,

we have to assign probability to each possible combination of our features.

So there are exponentially many combinations of different university grades,

different IQ scores and stuff like that.

And for each of them we have to assign a probability.

And this tables of probability

has been billions of entries and it's just impractical to treat as parameters,

to treat these probabilities as parameters.

So, we have to do something else but we always can assume some parametric model, right?

We can say that we have these five random variables

and that probability for any combination of them is some simple function.

For example, exponent of linear function divided by normalization constant.

In this case, you reduce your model complexity by a lot.

Now, we have just like five parameters which we want to train.

But the problem here is with the normalization constant.

So, to normalize this thing,

so it will be a proper probability and it will sum up to one,

we have to consider the normalization constant which

is the sum of all possible configurations.

And this is a gigantic sum.

We have to consider all billions of possible configurations to

compute it and this means that the training and inference will be impractical.

So what else can we do here?

Well, it turns out that you can introduce a new variable

which you don't actually have in your model of the world,

which is called intelligence.

So you can assume that each person has

some internal and hidden property of him

which we will call intelligence and for example measure on the scale from one to 100.

This intelligence directly causes each of

these IQ scores and university grades and stuff like that.

Of course, this connection is non-deterministic.

So an intelligent person can have a bad day and write test poorly.

But this is direct causation,

so intelligence directly causes all these observations.

And if I assume such a model,

then we reduce the model complexity by a lot.

We raised lots of features and now our model is much simpler to work with.

So now, we're going to write

our probabilistic model by using the rule of sum of probabilities,

it's the sum of

all possible configurations given the intelligence times the prior probability.

And these are conditional probability

factorizes into product of small probabilities because of the structure of our model.

So now, instead of one huge table with all the combinations for five different features,

we have just five small tables that assigns probabilities

to a pair of like IQ score given intelligence.

This means that they're able to reduce the model complexity and now to model

without reducing the flexibility of them all.

So to summarize, introducing latent variables may simplify our model.

So it can reduce the number of phases we have.

And as a consequence of that,

we can reduce the number of parameters.

And some other positive feature of latent variables,

is that they are sometimes interpretable.

So for example, these intelligence variable, we can,

for a new person we can estimate his intelligence on the scale from

one to 100 and for example it can be 80.

What does that mean?

Well, it's not obvious because you don't know

what the scale means and you're not even sure that

this intelligence means actual intelligence

because they never told you model that these variables should be intelligence,

you just said that there should be some variable here.

But anyway, this variable can be interpretable and you can compare

intelligence according to this scale of different people in your data set.

And some downside of latent variable models is that they can be harder to work with.

So, training latent variable model,

you have to rely on a lot math.

And this math is,

what this week is all about.

So in the next videos,

we will discuss methods for training latent variable models.

Ознакомьтесь с нашим каталогом

Присоединяйтесь бесплатно и получайте персонализированные рекомендации, обновления и предложения.

Coursera делает лучшее в мире образование доступным каждому, предлагая онлайн-курсы от ведущих университетов и организаций.

© Coursera Inc., 2018 Все права защищены.

Загрузить из App Store

Загрузить в Google Play