This course will cover the major techniques for mining and analyzing text data to discover interesting patterns, extract useful knowledge, and support decision making, with an emphasis on statistical approaches that can be generally applied to arbitrary text data in any natural language with no or minimum human effort. Detailed analysis of text data requires understanding of natural language text, which is known to be a difficult task for computers. However, a number of statistical approaches have been shown to work well for the "shallow" but robust analysis of text data for pattern finding and knowledge discovery. You will learn the basic concepts, principles, and major algorithms in text mining and their potential applications.
2.10 Probabilistic Topic Models: Mining One Topic
Loading...
Из курса от партнера University of Illinois at Urbana-Champaign
Text Mining and Analytics
316 оценки
Из урока
Week 2
During this module, you will learn more about word association mining with a particular focus on mining the other basic form of word association (i.e., syntagmatic relations), and start learning topic analysis with a focus on techniques for mining one topic from text.
Познакомьтесь с преподавателями
ChengXiang ZhaiProfessor
Department of Computer Science
[SOUND] This lecture is a continued
discussion of probabilistic topic models.
In this lecture, we're going to continue discussing probabilistic models.
We're going to talk about a very simple case where we
are interested in just mining one topic from one document.
So in this simple setup, we are interested in analyzing
one document and trying to discover just one topic.
So this is the simplest case of topic model.
The input now no longer has k, which is the number of topics because we
know there is only one topic and the collection has only one document, also.
In the output, we also no longer have coverage because
we assumed that the document covers this topic 100%.
So the main goal is just to discover the world of probabilities for
this single topic, as shown here.
As always, when we think about using a generating model to solve such a problem,
we start with thinking about what kind of data we are going to model or
from what perspective we're going to model the data or data representation.
And then we're going to design a specific model for
the generating of the data, from our perspective.
Where our perspective just means we want to take a particular angle of looking at
the data, so that the model will have the right parameters for
discovering the knowledge that we want.
And then we'll be thinking about the microfunction or
write down the microfunction to capture more formally how likely
a data point will be obtained from this model.
And the likelihood function will have some parameters in the function.
And then we argue our interest in estimating those parameters for example,
by maximizing the likelihood which will lead to maximum likelihood estimated.
These estimator parameters will then become the output
of the mining hours, which means we'll take the estimating
parameters as the knowledge that we discover from the text.
So let's look at these steps for this very simple case.
Later we'll look at this procedure for some more complicated cases.
So our data, in this case is, just a document which is a sequence of words.
Each word here is denoted by x sub i.
Our model is a Unigram language model.
A word distribution that we hope to denote a topic and that's our goal.
So we will have as many parameters as many words in our vocabulary, in this case M.
And for convenience we're going to use theta sub i to
denote the probability of word w sub i.
And obviously these theta sub i's will sum to 1.
Now what does a likelihood function look like?
Well, this is just the probability of generating this whole document,
that given such a model.
Because we assume the independence in generating each word so the probability of
the document will be just a product of the probability of each word.
And since some word might have repeated occurrences.
So we can also rewrite this product in a different form.
So in this line, we have rewritten the formula into a product
over all the unique words in the vocabulary, w sub 1 through w sub M.
Now this is different from the previous line.
Well, the product is over different positions of words in the document.
Now when we do this transformation, we then would need to
introduce a counter function here.
This denotes the count of word one in document and
similarly this is the count of words of n in the document
because these words might have repeated occurrences.
You can also see if a word did not occur in the document.
It will have a zero count, therefore that corresponding term will disappear.
So this is a very useful form of
writing down the likelihood function that we will often use later.
So I want you to pay attention to this, just get familiar with this notation.
It's just to change the product over all the different words in the vocabulary.
So in the end, of course, we'll use theta sub i to express this likelihood
function and it would look like this.
Next, we're going to find the theta values or probabilities
of these words that would maximize this likelihood function.
So now lets take a look at the maximum likelihood estimate problem more closely.
This line is copied from the previous slide.
It's just our likelihood function.
So our goal is to maximize this likelihood function.
We will find it often easy to
maximize the local likelihood instead of the original likelihood.
And this is purely for mathematical convenience because after
the logarithm transformation our function will becomes a sum instead of product.
And we also have constraints over these these probabilities.
The sum makes it easier to take derivative, which is often needed for
finding the optimal solution of this function.
So please take a look at this sum again, here.
And this is a form of a function that you will often
see later also, the more general topic models.
So it's a sum over all the words in the vocabulary.
And inside the sum there is a count of a word in the document.
And this is macroed by the logarithm of a probability.
So let's see how we can solve this problem.
Now at this point the problem is purely a mathematical problem because we are going
to just the find the optimal solution of a constrained maximization problem.
The objective function is the likelihood function and
the constraint is that all these probabilities must sum to one.
So, one way to solve the problem is to use Lagrange multiplier approace.
Now this command is beyond the scope of this course but
since Lagrange multiplier is a very useful approach, I also would like
to just give a brief introduction to this, for those of you who are interested.
So in this approach we will construct a Lagrange function, here.
And this function will combine our objective function
with another term that encodes our constraint and
we introduce Lagrange multiplier here,
lambda, so it's an additional parameter.
Now, the idea of this approach is just to turn the constraint optimization into,
in some sense, an unconstrained optimizing problem.
Now we are just interested in optimizing this Lagrange function.
As you may recall from calculus, an optimal point
would be achieved when the derivative is set to zero.
This is a necessary condition.
It's not sufficient, though.
So if we do that you will see the partial derivative,
with respect to theta i here ,is equal to this.
And this part comes from the derivative of the logarithm function and
this lambda is simply taken from here.
And when we set it to zero we can
easily see theta sub i is related to lambda in this way.
Since we know all the theta i's must a sum to one
we can plug this into this constraint, here.
And this will allow us to solve for lambda.
And this is just a net sum of all the counts.
And this further allows us to then solve the optimization problem,
eventually, to find the optimal setting for theta sub i.
And if you look at this formula it turns out that it's actually very intuitive
because this is just the normalized count of these words by the document ns,
which is also a sum of all the counts of words in the document.
So, after all this mess, after all,
we have just obtained something that's very intuitive and
this will be just our intuition where we want to
maximize the data by assigning as much probability
mass as possible to all the observed the words here.
And you might also notice that this is the general result of maximum likelihood
raised estimator.
In general, the estimator would be to normalize counts and it's just sometimes
the counts have to be done in a particular way, as you will also see later.
So this is basically an analytical solution to our optimization problem.
In general though, when the likelihood function is very complicated, we're not
going to be able to solve the optimization problem by having a closed form formula.
Instead we have to use some numerical algorithms and
we're going to see such cases later, also.
So if you imagine what would we get if we use such a maximum
likelihood estimator to estimate one topic for a single document d here?
Let's imagine this document is a text mining paper.
Now, what you might see is something that looks like this.
On the top, you will see the high probability words tend to be those very
common words, often functional words in English.
And this will be followed by some content words that really
characterize the topic well like text, mining, etc.
And then in the end, you also see there is more probability of
words that are not really related to the topic but
they might be extraneously mentioned in the document.
As a topic representation, you will see this is not ideal, right?
That because the high probability words are functional words,
they are not really characterizing the topic.
So my question is how can we get rid of such common words?
Now this is the topic of the next module.
We're going to talk about how to use probabilistic models to somehow get rid of
these common words.
[MUSIC]
Ознакомьтесь с нашим каталогом
Присоединяйтесь бесплатно и получайте персонализированные рекомендации, обновления и предложения.
Coursera делает лучшее в мире образование доступным каждому, предлагая онлайн-курсы от ведущих университетов и организаций.
© Coursera Inc., 2018 Все права защищены.
