Loading...

H.261, H.263, MPEG-1 and MPEG-2

Course video 60 of 72

In this module we discus video compression with an emphasis on motion-compensated hybrid video encoding and video compression standards including H.261, H.263, H.264, H.265, MPEG-1, MPEG-2, and MPEG-4.

Основы обработки цифровых видео и изображений

Северо-Западный университет

In this class you will learn the basic principles and tools used to process images and videos, and how to apply them in solving practical problems of commercial and scientific interests. Digital images and videos are everywhere these days – in thousands of scientific (e.g., astronomical, bio-medical), consumer, industrial, and artistic applications. Moreover they come in a wide range of the electromagnetic spectrum - from visible light and infrared to gamma rays and beyond. The ability to process image and video signals is therefore an incredibly important skill to master for engineering/science students, software developers, and practicing scientists. Digital image and video processing continues to enable the multimedia technology revolution we are experiencing today. Some important examples of image and video processing include the removal of degradations images suffer during acquisition (e.g., removing blur from a picture of a fast moving car), and the compression and transmission of images and videos (if you watch videos online, or share photos via a social media website, you use this everyday!), for economical storage and efficient transmission. This course will cover the fundamentals of image and video processing. We will provide a mathematical framework to describe and analyze images and videos as two- and three-dimensional signals in the spatial, spatio-temporal, and frequency domains. In this class not only will you learn the theory behind fundamental processing tasks including image/video enhancement, recovery, and compression - but you will also learn how to perform these key processing tasks in practice using state-of-the-art techniques and tools. We will introduce and use a wide variety of such tools – from optimization toolboxes to statistical techniques. Emphasis on the special role sparsity plays in modern image and video processing will also be given. In all cases, example images and videos pertaining to specific application domains will be utilized.

Просмотреть программу курса

Получаемые навыки

Image Processing, Data Compression, Matlab

Преподаватели

Aggelos K. Katsaggelos
Joseph Cummings Professor

Текст видео

Hello, welcome back.

In this segment we cover H.261, H.263, MPEG-1, and MPEG-2.

These all have been very successful standards

although there exist newer, more powerful ones.

The reason for covering them as well, and not just showing the latest and the

bravest is that there's a lot of innovation

conveyed by them when they were first introduced.

In addition, it's important to see the

evolution of standards to realize and appreciate what

improvements in the technology contribute to the advanced

performance of every new generation of [INAUDIBLE] standards.

We show the level of sophistication of each of these coders, and the decisions

that need to be made for the encoding of each and every so-called macroblock.

We also describe how MPEG-2 is handling

interlaced video, the first standard to do so.

So, with that, let's proceed with this topic.

[BLANK_AUDIO].

H2621 was one of the earliest video compression

standards it was approved by ITU-T, in 1990.

It provided quite a bit of innovation,

based on which subsequent standards build upon.

The application in mind was video

telephony, and video conferencing over ISDN.

Therefore the rates are integer multiples of 64 kilobits per second.

At these rates, small resolution frames are used, either SIF or QSIF.

What we see here is that the luminous

component, the y component is at this resolution,

while the CBCR, the [UNKNOWN] components are sub-sampled

by two in each direction, as shown here.

On the progressive, video is accommodated and

the aspect ratio is fixed, four over three.

H.263 was introduced by ITU-T again five years later in 95.

Now the application is video transmission over

wireless networks and public switched telephone networks.

So because of that, the range of rates is smaller than was .261.

And smaller resolutions are accommodated.

QCIF, but also sub-QCIF.

And the same story goes here, that the

chrominance components are sub-sampled versions of the luminance components.

And the progressive also video is addressed,

and a fixed four by three aspect ratio.

Some of the major differences between H.263 and H.261 are discussed here.

With 263, we can now achieve bitrates

below 64 kilobits per second, of course we're

dealing with QSIF and even sub- QSIF

resolutions and the frame rate is below 30.

Can be seven and a half frames per second or ten frames per second.

Unrestricted motion vectors are used, so the search area is not constrained.

And various advance motion conversation mode.

With 263, half pel motion vectors are estimated, while

with H.261 only integer pel motion vectors were used.

The loop filter is a filter that comes and go so it was introduced in 263, then

was not used for some of the standards, came back in H.265 for example.

So this is a filter that smooths out the prediction that is made

so it's in the feedback loop there of, of the predictor.

And also accommodates overlapping block motion compensation.

It covers four different types of frames, I stands for intra, P for

predictive or inter, B for bi directional, so prediction is perform

from the past as well as the future, and we cover these things back

in week four when we were talking about motion estimation.

And PB is a combination of predictive and by directional frames.

By and large these frames based on

my understanding have not really been extremely useful.

263 as compared with 261 satisfies the rule of thumb which is

it pretty much it achieves the same performance at half the bit rate.

Because of the contributions from MPEG, MPEG-1 was introduced

in 92 with the main application in mind that coding

of moving pictures and associated audio for storage applications

for up to one and a half megabits per second.

Two years later, MPEG-2 was introduced with a generic

application of coding of moving pictures and associated audio.

As I already mentioned, out of MPEG, came also MPEG-4 in 99, and to talk about it.

And jointly with ITU-T H.264 and H.265 became standards.

There are however, two more standards out of MPEG, MPEG-7

and MPEG-21 which are not video compression standards.

MPEG-7 standardizes the features that can be used for search by a

search engine, and MPEG-21 standardizes the seamless using,

user experience as the user moves around different platforms.

I've got to discuss the structure of the MPEG-1

video bitstream since it sheds some lights into the

inner workings of the coder, and also it's followed

with some modifications of course by almost all standards.

The basic building unit of any code there

is the block, its size eight by eight pixels.

Blocks are combined to form the macroblock layer.

The luminous component consists of four blocks.

While the CBCR, the chrominance components are sub sample

by two in each direction, therefore consists of two blocks.

Therefore there's six blocks for macroblock layer.

Macroblocks are then combined to give rise to the slice layer.

The idea of a slice is that this

represents an independently decodeable unit within the frame.

So we cut all dependencies from neighbors, and it can be therefore decode by itself.

In MPEG-1 slices have this form here.

It's all macroblocks.

In other standards that you'll see, slices can have an arbitrary shape.

So this many slices give rise to the picture layer.

We very often refer to this as a frame.

However, picture is a more general term because we may have interlaced for

example, video, and we can have frames and fields in that particular case.

Pictures are combined to give rise to the group of pictures layer or the GOP layer.

Within MPEG-1 the structure is fixed of the GOP.

So we have an intraframe, we have a predictive, or interframe,

and in between we have two bidirectionally encoded frame.

And then GOPs are combined to give rise to the sequence layer.

So with this structure it's rather straightforward to indicate

where a block belongs to which macroblock, to which slice layer,

to which picture, to which GOP and to which sequence layer.

I want to discuss here the very many

different options that are available in encoding a

macroblock, or the very many different decisions that

need to be made by the rate controller.

So if we deal with an I picture, intraframe picture, then all

the macroblock within this picture are also I macroblocks.

And in that case, the decision that needs to be made is whether to change

the step of the uniform quantizer, MQUANT or leave it the same.

And actually within the standards this change cannot be arbitrary.

It cannot go from quantization step two to

32, as an example, but it's done gradually.

When we deal with a P picture, an interpicture or a predictive picture, then

as we can see here, there are additionally many more decisions to be made.

So at the first level, the question is whether to

perform motion compensation, or set the motion vector equal to zero.

After motion compensation is performed, still we can decide whether

to treat this as an inter or an intra macro block.

So if the prediction error is very large, then it's decided

that this macroblock should be encoded as an intra one.

And then the decisions are whether to change

the quantizer step size or leave it the same.

When the decision is made to encode this macroblock as an inter macroblock, still

there are two options, we can choose to code the prediction error, or not code it.

If the prediction error is very small then just

doing motion compensation like pointing in the right direction

of the macroblock is good enough, and no error

is sent, it's very small, it's close to zero.

Now, if the prediction error is coded, then we still have to

decide whether to change or leave the same the quantize as step size.

If the decision is made to set the motion

vector equal to zero, then we'll still find the

prediction error and the same level of decision is

made whether to encode the macroblock as inter or intra.

If intra, then we have to decide on the quantize or step size, step size.

If inter, we have to decide whether to code or not code the prediction error.

If the decision is not to code it, then this is the skipped mode.

So zero motion vector and zero prediction error.

So in other words the current macroblock is just replaced by

the macroblock in the previous frame in exactly the same location.

If the predictioner is coded, then the decision of whether to

change the step size of the uniform quantizer needs to be made.

When we deal with a bidirectional picture, then

at the first level the decision has to

be made whether a forward or a backward

or a combination interpolated motion compensation should takes place.

And then for each of these decisions their subsequent step

is whether to encode the macroblock as inter or intra.

If intra we have to decide on the step size

of the uniform quantizer, if inter, you have decide whether

to code the prediction error or not code it, and

then decide again if coded on the quantizer step size.

So we clearly see that for each and every

macroblock, a multitude of decisions need to be made,.

And this is the job of the rate cont, rate controller, and these decisions

should be made in the array distortion optim out fashion, which means that for a

given bitrate, we should allocate the available

bits to encode the motion vectors and the

prediction error, so that the quality of

the reconstructed video is as high as possible.

We show here the bit distribution among the various

picture, or frames, I use, in the interchangeably, types.

So this is for the encoding of an

SIF video at 1.15 megabits per second using MPEG-1.

On the horizontal axis you see the frame number, while on

the vertical axis you see size of each picture or frame.

So on the average the I frames consume 156 kilobits, the

P frames 62 kilobits, and the B frames 15 kilobits.

So we see that there's an order of magnitude fewer bits in going from

I to B, and it's a reduction by more than 50% in going from I to P.

This is what we expect.

More bits are required for I frames that's you

might say the disadvantages compared to B and P frames.

On the other hand, the advantage of the I frames is that they stop error propagation

since they're decodable by themselves, and those who allow us to access the content.

If for example, we want to fast forward through an encoded digital movie then we

can only reconstruct the I frames and this

will provide this functionality of a digital VCR.

MPEG-1 supports a specific structure of the group of pictures.

It consists of 16 pictures, where the first one is an I frame, its followed

by two B's, a P, two B's, and so on, and this is exactly what we see here.

So although the specific numbers have to do with the

specific sequence that was encoded using MPEG-1, probably it goes

without saying that it's the typical profile that we expect

to see no matter which sequence, which movie we encode.

That is again, we need to spend most of our bits in

decoding an I frame, followed by a P, followed by a B.

When bidirectionally predicted frames are utilized, then there

are some issues that need to be addressed.

So we see the three different types of pictures, I, B and P.

What's important to notice is that P pictures are

predicted either from an I picture or from another P picture.

So this P one is predicted from another P.

On the other hand, B pictures are only allowed

to use either I or P pictures for the prediction.

So this B one is utilizing one I and one P picture, while this B for

example is utilizing one P from this side and one P from this side.

So because of this constrain, if we number the

frames it's clear that the order of encoding is one,

four, two, three, seven, five, six, and so on.

While clearly we want to play them back at the natural order.

One, two, three, four, five, six, seven.

So because of that there is some buffering, some delay that needs

to be introduced during encoding, decoding, and playing back.

To give a taste of how the macroblocks are distributed in encoding

a sequence by MPEG-1 we have the three different picture types here, so

when the picture is intra, then clearly all the macroblocks in it are intra.

However, when the picture type is P then some

macroblocks are intra coded, some are predictive, many more of

them intra, for some the motion vector is zero but

the prediction error is encoded and for some they're skipped.

Zero motion vector, zero prediction error.

Similarly for the bidirectionally encoded picture types there's some I

macroblocks, B macroblocks, bidirectional macroblocks and skip macroblocks.

And then the corresponding number of eight by eight blocks that were coded, right?

Here is it's intra then all these are coded.

And since one macroblock equals six blocks, then if you multiply

this times six, we're going to find the number of blocks coded.

And we have similar calculations along the way, created for the skipped

ones, there are no motion vectors no prediction error that is encoded.

We show here an example of the

distribution of the computational load in MPEG-1 decoding.

What we see and, it quite surprising maybe to some extent, is that the majority of

the computational load, like almost 40%, is due to motion compensation.

So just compensation, no estimation, the motion vectors are send

over by the encoder takes, 40% of the computation alone.

So one can imagine that if motion estimation was part

of it as in encoding, then this percentage will increase considerably.

Then, it's equally, almost distributed, by 20% to Huffman decoding,

inverse eight by eight DCTs and color transformation and display.

MPEG-2 was introduced two years after MPEG-1.

Although conceptually similar to MPEG-1, it includes

extensions to cover wider range of applications.

The main application is the broadcasting of TV quality

video at bitrates between four and nine megabits per second.

The syntax was found appropriate for other applications, such as HDTV,

which was the result of the formation of the grand alliance.

Some of the characteristics of MPEG-2 is

that it is backwards compatible with MPEG-1,

so if you have already an MPEG-1 decoder, it can decode an MPEG-2 bitstream.

It also handles, I,P-and B type of pictures as MPEG-1.

One of the significant enhancements is that now it can handle interlaced video.

And to say a few things about this.

It adopts this tool like approach, so is divided into profiles and levels.

And finally it can produce scalable bitstreams.

The idea of scalability is that the video is encoded once and then from

this encoded bitstream we can extract sub bitstreams, parts of it.

And you can construct the video, either at the lower

quality, if it's a PSNR scalable bitstream or at the lower

special resolution, and they are talking about special scalability or

at the lower frame rate, and they're talking about temporal scalability.

Interlaced video is a technique for doubling the perceived

frame rate of a video display without consuming extra bandwidth.

The interlace signal contains two fields, a top field and a

bottom field as show here, or an odd and an even field.

This enhances motion perception to the viewer, and reduces

[UNKNOWN] error, by taking into account the so called FI

phenomenon effect, which is the optical illusion of perceiving continuous

motion between separated motions when they viewed rapidly in succession.

So this effectively doubles their time

resolution or the temple resolution as compares

to, compare to non-interlaced video for frame rates that are equal to field rates.

So interlaced signals require a display that is capable

of showing the individual fields in a sequential order.

And only CRT displays and some plasma displays are capable of doing so.

So, because CRTs are not in existence anymore, most displays are progressive

ones, interlaced video becomes less and less important as time goes on.

So in the special representation we see here the two

fields that I mentioned the top field and the bottom field.

Over here we see the progressive scans and in the time domain here is the frame rate,

so for NTSC this is one 30th of a second, the distance between two frames.

Why when we deal with interlaced video the distance

between fields now is one 60th of a second.

When we deal with I frames we need to take

the DCT of eight by eight blocks of intensity value.

So given a 16 by 16 interlaced macroblock we have

two ways to break it into four, eight by eight blocks.

The first one shown here is the field mode.

[UNKNOWN] the 16 by 16 macroblock is broke into

these four eight by eight blocks, where here we

put the pixels that belong to field one, which

is also called the odd field or the top field.

While in these two blocks we put the pixels that belong

to the second field or the even field or the bottom field.

Clearly if the frame mode is chosen then this 16 by 16 interlace macroblock

is broken into four, eight by eight, also interlaced blocks as shown here.

Clearly this is a rate control decision but needs to be made.

MPEG-2 supports both the frame and the

field prediction mode for motion estimation compensation.

So, according to the frame prediction, the two fields are treated as a frame.

It's an interlaced frame and the motion

estimation compensation is done at the frame level.

In the field prediction mode we have an example here

where this field could be a top field or a

bottom field, and can be predicted from either the top

field or the bottom field of the previous frame in time.

While in this example, we're looking at the bottom field of

this particular frame at time t, let's say, which can be predicted

from the first field of the same frame, the same time instance,

or the bottom field of the previous field, times t minus one.

So in this case, the distance here is one 30th of a

second, while the distance between these two fields is one60th of a second.

Another motion compensation mode provided in MPEG-2 is the dual prime mode.

This mode is applicable only to P pictures and to GOPs that

have no B pictures between these P pictures and the reference frames.

So for frame pictures, two motion vectors are associated with each field.

So for this top field here, the motion vector v1 and

pv1 are associated with this, this dotted line.

If we, if we denote by pv1 X, the horizontal component

of this vector, then pv1 X is encoded in terms of v1x.

And, as you can see here, V1x connects this

top fields which are one 30th of a second apart.

While pv1 corrects the bottom field of this frame and the top

field of this one, which are one 60th of a frame apart.

Therefore, pv1 is half v1x, shown here.

And delta of x is a correction there.

If you can see there, the vertical component of pv1.

It's the same course and duration.

It's half of the v1 of y.

Plus, minus this term here, which corrects, takes into account

the shift between a top field and the bottom field.

An even field and an odd field.

And here is the correction term.

So, what's included in the bitstream is not

pv1 and pv2, but instead this correction factor delta.

Similar considerations are made here, when we have the, the field picture.

This table summarizes the key characteristics of the profiles and levels

defined for both scalable and non-scalable MPEG-2 to bitstreams.

In this table the upper bounds for the

picture resolution, frame rate and bitrates are provided.

The bitrate data refers to the maximum compressed

bitrate supported by the input buffers of the decoder.

While NA indicates that, that there

is no conformance restrictions for these variables.

It's expected that any implementations of the standard will

support only the main profiles at the main level.

The main profile provides the basic functionality of MPEG-2.

The simple profile is a low-cost version of the main profile.

Coding is similar, as in the main, but without any bidirectional prediction.

Levels refer to picture resolution.

The upper bound of the main level is C, CCIR, 601 picture format.

The high 1440 and the high level

resolutions correspond to picture resolutions for HDTV.

The low level resolution corresponds to SIF format.

So, these profiles and levels have a hierarchical relationship.

That is, a syntax supported by higher profile level

includes all of the elements of lower profile levels.

A decoder that can decode bitstreams at a certain profile level should also

be able to decode bitstreams at the lower profile level than it's own.

We show here some of the systems supporting HDTV around the world.

So in North America the result of the grand alliance goes

to adopt for video encoding MPEG-2 main profile at high level.

This is the aspect ratio, the similar one with movies the

special resolution and then for audio, Dolby Advanced Coding three is used.

In Europe there is digital video broadcast.

It also adopted MPEG-2, but the simple profile at high 1440.

The aspect ratio is different.

The resolution is different, and for audio encoding MPEG layer two is chosen.

And finally in Japan, there is Muse, is

using the same aspect ratio as North America, the

resolution is slightly different, but the video is

also similar to main profile at high level MPEG-2.

О Coursera

На онлайн-курсах, специализациях и дипломных программах у вас будут первоклассные преподаватели из лучших университетов и учебных заведений мира.

Join a community of 40 million learners from around the world

Earn a skill-based course certificate to apply your knowledge

Gain confidence in your skills and further your career

Coursera делает лучшее в мире образование доступным каждому, предлагая онлайн-курсы от ведущих университетов и организаций.

© Coursera Inc., 2019 Все права защищены.

Загрузить из App Store

Загрузить в Google Play