Introduction to Machine Translation

Loading...

From the course by National Research University Higher School of Economics

Natural Language Processing

124 оценки

National Research University Higher School of Economics

Natural Language Processing

124 оценки

Course 6 of 7 in the Specialization Advanced Machine Learning

This course covers a wide range of tasks in Natural Language Processing from basic to advanced: sentiment analysis, summarization, dialogue state tracking, to name a few. Upon completing, you will be able to recognize NLP tasks in your day-to-day work, propose approaches, and judge what techniques are likely to work well. The final project is devoted to one of the most hot topics in today’s NLP. You will build your own conversational chat-bot that will assist with search on StackOverflow website. The project will be based on practical assignments of the course, that will give you hands-on experience with such tasks as text classification, named entities recognition, and duplicates detection. Throughout the lectures, we will aim at finding a balance between traditional and deep learning techniques in NLP and cover them in parallel. For example, we will discuss word alignment models in machine translation and see how similar it is to attention mechanism in encoder-decoder neural networks. Core techniques are not treated as black boxes. On the contrary, you will get in-depth understanding of what’s happening inside. To succeed in that, we expect your familiarity with the basics of linear algebra and probability theory, machine learning setup, and deep neural networks. Some materials are based on one-month-old papers and introduce you to the very state-of-the-art in NLP research.

From the lesson

Sequence to sequence tasks

Nearly any task in NLP can be formulates as a sequence to sequence task: machine translation, summarization, question answering, and many more. In this module we will learn a general encoder-decoder-attention architecture that can be used to solve them. We will cover machine translation in more details and you will see how attention technique resembles word alignment task in traditional pipeline.

Introduction to Machine Translation15:30

Noisy channel: said in English, received in French6:08

Word Alignment Models12:28

Meet the Instructors

Anna Potapenko
Researcher
HSE Faculty of Computer Science
Alexey Zobnin
Accosiate professor
HSE Faculty of Computer Science
Anna Kozlova
Team Lead
Yandex
Sergey Yudin
Analyst-developer
Yandex
Andrei Zimovnov
Senior Lecturer
HSE Faculty of Computer Science

[MUSIC]

Hi everyone, this week is about sequence to sequence tasks.

We have a lot of them in NLP, but

one obvious example would be machine translation.

So you have a sequence of words in one language as an input, and

you want to produce a sequence of words in some other language as an output.

Now, you can think about some other examples.

For example, summarization is also a sequence to sequence task and

you can think about it as machine translation but for

one language, monolingual machine translation.

Well we will cover these examples in the end of the week but now let us start with

statistical machine translation, and neural machine translation.

We will see that actually there are some techniques,

that are super similar in both these approaches.

For example, we will see alignments,

word alignments that we need in statistical machine translation.

And then, we will see that we have attention mechanism in neural networks

that kind of has similar meaning in these tasks.

Okay, so let us begin, and I think there is no need to tell you that

machine translation is important, we just know that.

So I would better start with two other questions.

Two questions that actually we skip a lot in our course and

in some other courses but these are two very important questions to speak about.

So one question is data and another question is evaluation.

When you get some real task in your life, some NLP task usually

this is not a model that is plane, this is usually data and evaluation.

So you can have a fancy neuro-architecture, but

if you do not have good data and if you haven't settled

down how to do evaluation procedure, you're not going to have good results.

So first data, well what kind of data do we need for machine translation?

We need some parallel corpora, so we need some text in one language and

we need its translation to another language.

Where does that come from, so what sources can you think of?

Well, one of your source well maybe not so obvious but

one very good source, is European Parliament proceedings.

So you have there some texts in several languages, maybe 20 languages and

very exact translations of one in the same statements.

And this is nice, so you can use that, some other domain would be movies.

So you have subtitles that are translated in many languages this is nice.

Something which is not that useful, but still useful,

would be books translations or Wikipedia articles.

So for example, for Wikipedia you can not guarantee that you

have the same text for two languages.

But you can have something similar, for example some vague translations or

which are to the same topic at least.

So we call this corpora comparable but not parallel.

The OPUS website has the nice overview of many sources so please check it out.

But I want to discuss something which is not nice, some problems with the data.

Actually, we have lots of problems for any data that we have, and

what kind of problems happen for machine translation?

Well, first, usually the data comes from some specific domain.

So imagine you have movie subtitles and you want to train a system for

scientific papers translations.

It's not going to work, right, so you need to have some close domain.

Or you need to know how to transfer your knowledge from one domain

to another domain, this is something to think about.

Now, you can have some decent amount of data for some language pairs like English

and French, or English and German, but probably for some rare language pairs,

you have really not a lot of data, and that's a huge problem.

Also you can have noisy and not enough data, and it can be not aligned well.

By alignment I mean, you need to know the correspondence between the sentences,

or even better the correspondence between the words and the sentences.

And this is luxury, so usually you do not have that, at least for

a huge amount of data.

Okay, now I think it's clear about the data, so the second thing, evaluation.

Well you can say that we have some parallel data.

So why don't we just split it to train and test and have our test set

to compare correct translations and those that are produced by our system.

But well, how do we know that the translation

is wrong just because it doesn't occur in your reference?

You know that the language is so

relative so every translator would do some different translations.

It means that if your system produce something different it doesn't mean yet

that it is wrong.

So well there is no nice answer for this question, I mean this is a problem, yes.

One thing that you can do is to have multiple references so you can have,

let's say five references and compare your system output to all of them.

And the other thing is you should be very careful how do you compare it.

So definitely you shouldn't do just exact match,

right you should do something more intelligent.

And I'm going to show you BLUE score which is known to be very popular measure

in machine translation that try somehow to softly measure whether your

system output is somehow similar to the reference translation.

Okay, let me show you an example.

So you have some reference translation and you have the output of your system and

you try to compare them.

Well, you remember that we have this nice tool which is called engrams.

So you can compute some unigrams and bigrams and trigrams.

Do you have any idea how to use that here?

Well, first we can try to compute some precision, what does it mean?

You look into your system output, and here you have six words,

six unigrams and compute how many of them actually occur in the reference.

So the unigram precision core will be 4 out of 6.

Now, tell me what would be bigram score here.

Well, the bigram score will be 3 out of 5 because you have

5 bigrams in your system output and only 3 of them was sent sent on and

on Tuesday occurred in the reference.

Now you can proceed and you can compute 3-grams score and

4-grams score, so that's good.

Maybe we can just average them and have some measure.

Well we could, but there is one problem here,

well imagine that the system tries to be super precise.

Then it is good for system to output super short sentences, right?

So if I'm sure that this union gram should occur,

I will just output this and I will not output more.

So just to punish into penalty the model, we can have some brevity score.

This brevity penalty says that we

divide the length of the output by the length of the reference.

And then the system outputs two short sentences, we will get to know that.

Now how do we compute the BLEU score out of these values?

Like this so we have some average so this root is the average

of our union gram, bigram, 3-gram, and 4-gram's course.

And then we multiply this average by the brevity.

Okay, now let us speak about how the system actually works.

So this is kind of a mandatory slide on machine translation,

because kind of any tutorial on machine translation has this.

So I decided not to be an exception and show you that.

So the idea is like that, we have some source sentence and

we want to translate it to get some target sentence.

Now the first thing that we can do is just direct transfer.

So we can translate this source sentence word by word and get the target sentence.

But well, maybe it's not super good, right?

So if you have ever studied some foreign language, you know that just by dictionary

translations of every word, you usually do not get nice coherent translation.

So probably we would better go into some synthetic level.

So we do syntax analysis, and then we do the transfer and

then we generate the target sentence by knowing how it

should look like on on the syntactic level.

Even better, we could try to go to semantic layer, so that first we analyze

the source sentence and understand some meanings of some parts of the sentence.

We somehow transfer these meanings to in our language and

then we generate some good syntactic structures with good meaning.

And our dream, like the best things as we could ever think of,

would be having some interlingual.

So by interlingual, we mean some n ice representation of the whole

source sentence that is enough to generate the whole target sentence.

Actually it is still a dream, so it is still a dream of the translators

to have that kind of system because it sounds so appealing.

But neural translation systems somehow have mechanisms that

resembles that and I will show you that in a couple of slides.

Okay, so for now I want to show you some brief history of the area.

And like any other area,

machine translation has some bright and dark periods.

So in 1954 there were great expectations, so there was IBM experiments where

they translated 60 sentences from Russian to English.

And they said, that's easy we can solve the machine translation

task completely in just three or five years.

So they tried to work on that and they worked a lot, and

after many years they concluded that actually it's not that easy.

And they said, well, machine translation is too expensive and

we should not do automatic machine translation system.

We should better focus on just some tools that help human

translators to provide good quality translations.

So you know these great expectations and

then the disappointment made the area silent for

a while, but then in 1988 IBM researchers

proposed word-based machine translation systems.

These machine translation systems were rather simple, so

we will cover them, kind of in this video and in the next video, but

these systems were kind of the first working system for machine translation.

So this was nice and then the next important step was phrase based machine

translations system that were proposed by Philip Koehn in 2003.

And this is what probably people mean by statistical machine translation now.

You definitely know Google Translate, right?

But maybe you haven't heard about Moses.

So Moses is the system that allows a researchers to build their own

machine translation systems.

So it allows to train your models and to compare them, so

this is a very nice tool for researchers and it was made available in 2007.

Now, with an extent,

obviously very important step here is neural machine translation.

It is amazing how fast the neural machine translation systems

could go from research papers to production.

Usually we have such a big gap between these two things.

But in this case there were just two or three years so

it is amazing that those ideas that were proposed could be implemented and

just launched in many companies in 2016 so we have neutral machine translations now.

You might be wondering what is WMT there, it is the workshop on machine translation,

which is kind of the annual competition, the annual event and shared tasks.

Which means that you can compare your systems there, and it is a very nice

venue to compare different systems by different researchers and companies.

And to see what are the traits of machine translations.

And it happens every year, so usually people who do research in this

area keep eye on this and this is very nice thing.

This is the slide about intralingual that I promised to show you.

So this is how Google neural machine translation works, and

there was actually lots of hype around it maybe even too much.

But still, so the idea is that you train some system or some pair of languages.

For example on English to Japanese and Japanese to English and

English to Korean and some other pair, you train some encoder, decoder architecture.

It means that you have some encoder that encodes your sentence to

some hidden representation.

And then you have decoder that takes that hidden representation and

decodes it to the target sentence.

Now the nice thing is, that if you just take your encoder, let's say for

Japanese and decoder for Korean and you just take them.

Somehow it works nicely even though the system has never seen

Japanese to Korean translations.

You see so this is zero-shot translation you have never seen Japanese to Korean,

but just by building nice encoder and

nice decoder, you can stack them and get this path.

So it seems like this hidden representation that you have,

is kind of universal for any language pair.

Well, it is not completely true but at least it is very promising result.

[MUSIC]

Ознакомьтесь с нашим каталогом

Присоединяйтесь бесплатно и получайте персонализированные рекомендации, обновления и предложения.

Coursera делает лучшее в мире образование доступным каждому, предлагая онлайн-курсы от ведущих университетов и организаций.

© Coursera Inc., 2018 Все права защищены.

Загрузить из App Store

Загрузить в Google Play