2.10 Probabilistic Topic Models: Mining One Topic

Loading...

Из курса от партнера University of Illinois at Urbana-Champaign

Text Mining and Analytics

346 оценки

University of Illinois at Urbana-Champaign

Text Mining and Analytics

346 оценки

Курс 3 из 6 — Specialization Data Mining

This course will cover the major techniques for mining and analyzing text data to discover interesting patterns, extract useful knowledge, and support decision making, with an emphasis on statistical approaches that can be generally applied to arbitrary text data in any natural language with no or minimum human effort. Detailed analysis of text data requires understanding of natural language text, which is known to be a difficult task for computers. However, a number of statistical approaches have been shown to work well for the "shallow" but robust analysis of text data for pattern finding and knowledge discovery. You will learn the basic concepts, principles, and major algorithms in text mining and their potential applications.

Из урока

Week 2

During this module, you will learn more about word association mining with a particular focus on mining the other basic form of word association (i.e., syntagmatic relations), and start learning topic analysis with a focus on techniques for mining one topic from text.

2.1 Syntagmatic Relation Discovery: Entropy11:00

2.2 Syntagmatic Relation Discovery: Conditional Entropy11:57

2.3 Syntagmatic Relation Discovery: Mutual Information: Part 113:55

2.4 Syntagmatic Relation Discovery: Mutual Information: Part 29:42

2.5 Topic Mining and Analysis: Motivation and Task Definition7:36

2.6 Topic Mining and Analysis: Term as Topic11:31

2.7 Topic Mining and Analysis: Probabilistic Topic Models14:17

2.8 Probabilistic Topic Models: Overview of Statistical Language Models: Part 110:25

2.9 Probabilistic Topic Models: Overview of Statistical Language Models: Part 213:11

2.10 Probabilistic Topic Models: Mining One Topic12:21

Познакомьтесь с преподавателями

ChengXiang Zhai
Professor
Department of Computer Science

[SOUND] This lecture is a continued

discussion of probabilistic topic models.

In this lecture, we're going to continue discussing probabilistic models.

We're going to talk about a very simple case where we

are interested in just mining one topic from one document.

So in this simple setup, we are interested in analyzing

one document and trying to discover just one topic.

So this is the simplest case of topic model.

The input now no longer has k, which is the number of topics because we

know there is only one topic and the collection has only one document, also.

In the output, we also no longer have coverage because

we assumed that the document covers this topic 100%.

So the main goal is just to discover the world of probabilities for

this single topic, as shown here.

As always, when we think about using a generating model to solve such a problem,

we start with thinking about what kind of data we are going to model or

from what perspective we're going to model the data or data representation.

And then we're going to design a specific model for

the generating of the data, from our perspective.

Where our perspective just means we want to take a particular angle of looking at

the data, so that the model will have the right parameters for

discovering the knowledge that we want.

And then we'll be thinking about the microfunction or

write down the microfunction to capture more formally how likely

a data point will be obtained from this model.

And the likelihood function will have some parameters in the function.

And then we argue our interest in estimating those parameters for example,

by maximizing the likelihood which will lead to maximum likelihood estimated.

These estimator parameters will then become the output

of the mining hours, which means we'll take the estimating

parameters as the knowledge that we discover from the text.

So let's look at these steps for this very simple case.

Later we'll look at this procedure for some more complicated cases.

So our data, in this case is, just a document which is a sequence of words.

Each word here is denoted by x sub i.

Our model is a Unigram language model.

A word distribution that we hope to denote a topic and that's our goal.

So we will have as many parameters as many words in our vocabulary, in this case M.

And for convenience we're going to use theta sub i to

denote the probability of word w sub i.

And obviously these theta sub i's will sum to 1.

Now what does a likelihood function look like?

Well, this is just the probability of generating this whole document,

that given such a model.

Because we assume the independence in generating each word so the probability of

the document will be just a product of the probability of each word.

And since some word might have repeated occurrences.

So we can also rewrite this product in a different form.

So in this line, we have rewritten the formula into a product

over all the unique words in the vocabulary, w sub 1 through w sub M.

Now this is different from the previous line.

Well, the product is over different positions of words in the document.

Now when we do this transformation, we then would need to

introduce a counter function here.

This denotes the count of word one in document and

similarly this is the count of words of n in the document

because these words might have repeated occurrences.

You can also see if a word did not occur in the document.

It will have a zero count, therefore that corresponding term will disappear.

So this is a very useful form of

writing down the likelihood function that we will often use later.

So I want you to pay attention to this, just get familiar with this notation.

It's just to change the product over all the different words in the vocabulary.

So in the end, of course, we'll use theta sub i to express this likelihood

function and it would look like this.

Next, we're going to find the theta values or probabilities

of these words that would maximize this likelihood function.

So now lets take a look at the maximum likelihood estimate problem more closely.

This line is copied from the previous slide.

It's just our likelihood function.

So our goal is to maximize this likelihood function.

We will find it often easy to

maximize the local likelihood instead of the original likelihood.

And this is purely for mathematical convenience because after

the logarithm transformation our function will becomes a sum instead of product.

And we also have constraints over these these probabilities.

The sum makes it easier to take derivative, which is often needed for

finding the optimal solution of this function.

So please take a look at this sum again, here.

And this is a form of a function that you will often

see later also, the more general topic models.

So it's a sum over all the words in the vocabulary.

And inside the sum there is a count of a word in the document.

And this is macroed by the logarithm of a probability.

So let's see how we can solve this problem.

Now at this point the problem is purely a mathematical problem because we are going

to just the find the optimal solution of a constrained maximization problem.

The objective function is the likelihood function and

the constraint is that all these probabilities must sum to one.

So, one way to solve the problem is to use Lagrange multiplier approace.

Now this command is beyond the scope of this course but

since Lagrange multiplier is a very useful approach, I also would like

to just give a brief introduction to this, for those of you who are interested.

So in this approach we will construct a Lagrange function, here.

And this function will combine our objective function

with another term that encodes our constraint and

we introduce Lagrange multiplier here,

lambda, so it's an additional parameter.

Now, the idea of this approach is just to turn the constraint optimization into,

in some sense, an unconstrained optimizing problem.

Now we are just interested in optimizing this Lagrange function.

As you may recall from calculus, an optimal point

would be achieved when the derivative is set to zero.

This is a necessary condition.

It's not sufficient, though.

So if we do that you will see the partial derivative,

with respect to theta i here ,is equal to this.

And this part comes from the derivative of the logarithm function and

this lambda is simply taken from here.

And when we set it to zero we can

easily see theta sub i is related to lambda in this way.

Since we know all the theta i's must a sum to one

we can plug this into this constraint, here.

And this will allow us to solve for lambda.

And this is just a net sum of all the counts.

And this further allows us to then solve the optimization problem,

eventually, to find the optimal setting for theta sub i.

And if you look at this formula it turns out that it's actually very intuitive

because this is just the normalized count of these words by the document ns,

which is also a sum of all the counts of words in the document.

So, after all this mess, after all,

we have just obtained something that's very intuitive and

this will be just our intuition where we want to

maximize the data by assigning as much probability

mass as possible to all the observed the words here.

And you might also notice that this is the general result of maximum likelihood

raised estimator.

In general, the estimator would be to normalize counts and it's just sometimes

the counts have to be done in a particular way, as you will also see later.

So this is basically an analytical solution to our optimization problem.

In general though, when the likelihood function is very complicated, we're not

going to be able to solve the optimization problem by having a closed form formula.

Instead we have to use some numerical algorithms and

we're going to see such cases later, also.

So if you imagine what would we get if we use such a maximum

likelihood estimator to estimate one topic for a single document d here?

Let's imagine this document is a text mining paper.

Now, what you might see is something that looks like this.

On the top, you will see the high probability words tend to be those very

common words, often functional words in English.

And this will be followed by some content words that really

characterize the topic well like text, mining, etc.

And then in the end, you also see there is more probability of

words that are not really related to the topic but

they might be extraneously mentioned in the document.

As a topic representation, you will see this is not ideal, right?

That because the high probability words are functional words,

they are not really characterizing the topic.

So my question is how can we get rid of such common words?

Now this is the topic of the next module.

We're going to talk about how to use probabilistic models to somehow get rid of

these common words.

[MUSIC]

Ознакомьтесь с нашим каталогом

Присоединяйтесь бесплатно и получайте персонализированные рекомендации, обновления и предложения.

Coursera делает лучшее в мире образование доступным каждому, предлагая онлайн-курсы от ведущих университетов и организаций.

© Coursera Inc., 2019 Все права защищены.

Загрузить из App Store

Загрузить в Google Play