In this class you will learn the basic principles and tools used to process images and videos, and how to apply them in solving practical problems of commercial and scientific interests. Digital images and videos are everywhere these days – in thousands of scientific (e.g., astronomical, bio-medical), consumer, industrial, and artistic applications. Moreover they come in a wide range of the electromagnetic spectrum - from visible light and infrared to gamma rays and beyond. The ability to process image and video signals is therefore an incredibly important skill to master for engineering/science students, software developers, and practicing scientists. Digital image and video processing continues to enable the multimedia technology revolution we are experiencing today. Some important examples of image and video processing include the removal of degradations images suffer during acquisition (e.g., removing blur from a picture of a fast moving car), and the compression and transmission of images and videos (if you watch videos online, or share photos via a social media website, you use this everyday!), for economical storage and efficient transmission. This course will cover the fundamentals of image and video processing. We will provide a mathematical framework to describe and analyze images and videos as two- and three-dimensional signals in the spatial, spatio-temporal, and frequency domains. In this class not only will you learn the theory behind fundamental processing tasks including image/video enhancement, recovery, and compression - but you will also learn how to perform these key processing tasks in practice using state-of-the-art techniques and tools. We will introduce and use a wide variety of such tools – from optimization toolboxes to statistical techniques. Emphasis on the special role sparsity plays in modern image and video processing will also be given. In all cases, example images and videos pertaining to specific application domains will be utilized.
H.261, H.263, MPEG-1 and MPEG-2
Loading...
Из курса от партнера Northwestern University
Fundamentals of Digital Image and Video Processing
664 оценки
Из урока
Video Compression
In this module we discus video compression with an emphasis on motion-compensated hybrid video encoding and video compression standards including H.261, H.263, H.264, H.265, MPEG-1, MPEG-2, and MPEG-4.
Познакомьтесь с преподавателями
Aggelos K. KatsaggelosJoseph Cummings Professor
Department of Electrical Engineering and Computer Science
Hello, welcome back.
In this segment we cover H.261, H.263, MPEG-1, and MPEG-2.
These all have been very successful standards
although there exist newer, more powerful ones.
The reason for covering them as well, and not just showing the latest and the
bravest is that there's a lot of innovation
conveyed by them when they were first introduced.
In addition, it's important to see the
evolution of standards to realize and appreciate what
improvements in the technology contribute to the advanced
performance of every new generation of [INAUDIBLE] standards.
We show the level of sophistication of each of these coders, and the decisions
that need to be made for the encoding of each and every so-called macroblock.
We also describe how MPEG-2 is handling
interlaced video, the first standard to do so.
So, with that, let's proceed with this topic.
[BLANK_AUDIO].
H2621 was one of the earliest video compression
standards it was approved by ITU-T, in 1990.
It provided quite a bit of innovation,
based on which subsequent standards build upon.
The application in mind was video
telephony, and video conferencing over ISDN.
Therefore the rates are integer multiples of 64 kilobits per second.
At these rates, small resolution frames are used, either SIF or QSIF.
What we see here is that the luminous
component, the y component is at this resolution,
while the CBCR, the [UNKNOWN] components are sub-sampled
by two in each direction, as shown here.
On the progressive, video is accommodated and
the aspect ratio is fixed, four over three.
H.263 was introduced by ITU-T again five years later in 95.
Now the application is video transmission over
wireless networks and public switched telephone networks.
So because of that, the range of rates is smaller than was .261.
And smaller resolutions are accommodated.
QCIF, but also sub-QCIF.
And the same story goes here, that the
chrominance components are sub-sampled versions of the luminance components.
And the progressive also video is addressed,
and a fixed four by three aspect ratio.
Some of the major differences between H.263 and H.261 are discussed here.
With 263, we can now achieve bitrates
below 64 kilobits per second, of course we're
dealing with QSIF and even sub- QSIF
resolutions and the frame rate is below 30.
Can be seven and a half frames per second or ten frames per second.
Unrestricted motion vectors are used, so the search area is not constrained.
And various advance motion conversation mode.
With 263, half pel motion vectors are estimated, while
with H.261 only integer pel motion vectors were used.
The loop filter is a filter that comes and go so it was introduced in 263, then
was not used for some of the standards, came back in H.265 for example.
So this is a filter that smooths out the prediction that is made
so it's in the feedback loop there of, of the predictor.
And also accommodates overlapping block motion compensation.
It covers four different types of frames, I stands for intra, P for
predictive or inter, B for bi directional, so prediction is perform
from the past as well as the future, and we cover these things back
in week four when we were talking about motion estimation.
And PB is a combination of predictive and by directional frames.
By and large these frames based on
my understanding have not really been extremely useful.
263 as compared with 261 satisfies the rule of thumb which is
it pretty much it achieves the same performance at half the bit rate.
Because of the contributions from MPEG, MPEG-1 was introduced
in 92 with the main application in mind that coding
of moving pictures and associated audio for storage applications
for up to one and a half megabits per second.
Two years later, MPEG-2 was introduced with a generic
application of coding of moving pictures and associated audio.
As I already mentioned, out of MPEG, came also MPEG-4 in 99, and to talk about it.
And jointly with ITU-T H.264 and H.265 became standards.
There are however, two more standards out of MPEG, MPEG-7
and MPEG-21 which are not video compression standards.
MPEG-7 standardizes the features that can be used for search by a
search engine, and MPEG-21 standardizes the seamless using,
user experience as the user moves around different platforms.
I've got to discuss the structure of the MPEG-1
video bitstream since it sheds some lights into the
inner workings of the coder, and also it's followed
with some modifications of course by almost all standards.
The basic building unit of any code there
is the block, its size eight by eight pixels.
Blocks are combined to form the macroblock layer.
The luminous component consists of four blocks.
While the CBCR, the chrominance components are sub sample
by two in each direction, therefore consists of two blocks.
Therefore there's six blocks for macroblock layer.
Macroblocks are then combined to give rise to the slice layer.
The idea of a slice is that this
represents an independently decodeable unit within the frame.
So we cut all dependencies from neighbors, and it can be therefore decode by itself.
In MPEG-1 slices have this form here.
It's all macroblocks.
In other standards that you'll see, slices can have an arbitrary shape.
So this many slices give rise to the picture layer.
We very often refer to this as a frame.
However, picture is a more general term because we may have interlaced for
example, video, and we can have frames and fields in that particular case.
Pictures are combined to give rise to the group of pictures layer or the GOP layer.
Within MPEG-1 the structure is fixed of the GOP.
So we have an intraframe, we have a predictive, or interframe,
and in between we have two bidirectionally encoded frame.
And then GOPs are combined to give rise to the sequence layer.
So with this structure it's rather straightforward to indicate
where a block belongs to which macroblock, to which slice layer,
to which picture, to which GOP and to which sequence layer.
I want to discuss here the very many
different options that are available in encoding a
macroblock, or the very many different decisions that
need to be made by the rate controller.
So if we deal with an I picture, intraframe picture, then all
the macroblock within this picture are also I macroblocks.
And in that case, the decision that needs to be made is whether to change
the step of the uniform quantizer, MQUANT or leave it the same.
And actually within the standards this change cannot be arbitrary.
It cannot go from quantization step two to
32, as an example, but it's done gradually.
When we deal with a P picture, an interpicture or a predictive picture, then
as we can see here, there are additionally many more decisions to be made.
So at the first level, the question is whether to
perform motion compensation, or set the motion vector equal to zero.
After motion compensation is performed, still we can decide whether
to treat this as an inter or an intra macro block.
So if the prediction error is very large, then it's decided
that this macroblock should be encoded as an intra one.
And then the decisions are whether to change
the quantizer step size or leave it the same.
When the decision is made to encode this macroblock as an inter macroblock, still
there are two options, we can choose to code the prediction error, or not code it.
If the prediction error is very small then just
doing motion compensation like pointing in the right direction
of the macroblock is good enough, and no error
is sent, it's very small, it's close to zero.
Now, if the prediction error is coded, then we still have to
decide whether to change or leave the same the quantize as step size.
If the decision is made to set the motion
vector equal to zero, then we'll still find the
prediction error and the same level of decision is
made whether to encode the macroblock as inter or intra.
If intra, then we have to decide on the quantize or step size, step size.
If inter, we have to decide whether to code or not code the prediction error.
If the decision is not to code it, then this is the skipped mode.
So zero motion vector and zero prediction error.
So in other words the current macroblock is just replaced by
the macroblock in the previous frame in exactly the same location.
If the predictioner is coded, then the decision of whether to
change the step size of the uniform quantizer needs to be made.
When we deal with a bidirectional picture, then
at the first level the decision has to
be made whether a forward or a backward
or a combination interpolated motion compensation should takes place.
And then for each of these decisions their subsequent step
is whether to encode the macroblock as inter or intra.
If intra we have to decide on the step size
of the uniform quantizer, if inter, you have decide whether
to code the prediction error or not code it, and
then decide again if coded on the quantizer step size.
So we clearly see that for each and every
macroblock, a multitude of decisions need to be made,.
And this is the job of the rate cont, rate controller, and these decisions
should be made in the array distortion optim out fashion, which means that for a
given bitrate, we should allocate the available
bits to encode the motion vectors and the
prediction error, so that the quality of
the reconstructed video is as high as possible.
We show here the bit distribution among the various
picture, or frames, I use, in the interchangeably, types.
So this is for the encoding of an
SIF video at 1.15 megabits per second using MPEG-1.
On the horizontal axis you see the frame number, while on
the vertical axis you see size of each picture or frame.
So on the average the I frames consume 156 kilobits, the
P frames 62 kilobits, and the B frames 15 kilobits.
So we see that there's an order of magnitude fewer bits in going from
I to B, and it's a reduction by more than 50% in going from I to P.
This is what we expect.
More bits are required for I frames that's you
might say the disadvantages compared to B and P frames.
On the other hand, the advantage of the I frames is that they stop error propagation
since they're decodable by themselves, and those who allow us to access the content.
If for example, we want to fast forward through an encoded digital movie then we
can only reconstruct the I frames and this
will provide this functionality of a digital VCR.
MPEG-1 supports a specific structure of the group of pictures.
It consists of 16 pictures, where the first one is an I frame, its followed
by two B's, a P, two B's, and so on, and this is exactly what we see here.
So although the specific numbers have to do with the
specific sequence that was encoded using MPEG-1, probably it goes
without saying that it's the typical profile that we expect
to see no matter which sequence, which movie we encode.
That is again, we need to spend most of our bits in
decoding an I frame, followed by a P, followed by a B.
When bidirectionally predicted frames are utilized, then there
are some issues that need to be addressed.
So we see the three different types of pictures, I, B and P.
What's important to notice is that P pictures are
predicted either from an I picture or from another P picture.
So this P one is predicted from another P.
On the other hand, B pictures are only allowed
to use either I or P pictures for the prediction.
So this B one is utilizing one I and one P picture, while this B for
example is utilizing one P from this side and one P from this side.
So because of this constrain, if we number the
frames it's clear that the order of encoding is one,
four, two, three, seven, five, six, and so on.
While clearly we want to play them back at the natural order.
One, two, three, four, five, six, seven.
So because of that there is some buffering, some delay that needs
to be introduced during encoding, decoding, and playing back.
To give a taste of how the macroblocks are distributed in encoding
a sequence by MPEG-1 we have the three different picture types here, so
when the picture is intra, then clearly all the macroblocks in it are intra.
However, when the picture type is P then some
macroblocks are intra coded, some are predictive, many more of
them intra, for some the motion vector is zero but
the prediction error is encoded and for some they're skipped.
Zero motion vector, zero prediction error.
Similarly for the bidirectionally encoded picture types there's some I
macroblocks, B macroblocks, bidirectional macroblocks and skip macroblocks.
And then the corresponding number of eight by eight blocks that were coded, right?
Here is it's intra then all these are coded.
And since one macroblock equals six blocks, then if you multiply
this times six, we're going to find the number of blocks coded.
And we have similar calculations along the way, created for the skipped
ones, there are no motion vectors no prediction error that is encoded.
We show here an example of the
distribution of the computational load in MPEG-1 decoding.
What we see and, it quite surprising maybe to some extent, is that the majority of
the computational load, like almost 40%, is due to motion compensation.
So just compensation, no estimation, the motion vectors are send
over by the encoder takes, 40% of the computation alone.
So one can imagine that if motion estimation was part
of it as in encoding, then this percentage will increase considerably.
Then, it's equally, almost distributed, by 20% to Huffman decoding,
inverse eight by eight DCTs and color transformation and display.
MPEG-2 was introduced two years after MPEG-1.
Although conceptually similar to MPEG-1, it includes
extensions to cover wider range of applications.
The main application is the broadcasting of TV quality
video at bitrates between four and nine megabits per second.
The syntax was found appropriate for other applications, such as HDTV,
which was the result of the formation of the grand alliance.
Some of the characteristics of MPEG-2 is
that it is backwards compatible with MPEG-1,
so if you have already an MPEG-1 decoder, it can decode an MPEG-2 bitstream.
It also handles, I,P-and B type of pictures as MPEG-1.
One of the significant enhancements is that now it can handle interlaced video.
And to say a few things about this.
It adopts this tool like approach, so is divided into profiles and levels.
And finally it can produce scalable bitstreams.
The idea of scalability is that the video is encoded once and then from
this encoded bitstream we can extract sub bitstreams, parts of it.
And you can construct the video, either at the lower
quality, if it's a PSNR scalable bitstream or at the lower
special resolution, and they are talking about special scalability or
at the lower frame rate, and they're talking about temporal scalability.
Interlaced video is a technique for doubling the perceived
frame rate of a video display without consuming extra bandwidth.
The interlace signal contains two fields, a top field and a
bottom field as show here, or an odd and an even field.
This enhances motion perception to the viewer, and reduces
[UNKNOWN] error, by taking into account the so called FI
phenomenon effect, which is the optical illusion of perceiving continuous
motion between separated motions when they viewed rapidly in succession.
So this effectively doubles their time
resolution or the temple resolution as compares
to, compare to non-interlaced video for frame rates that are equal to field rates.
So interlaced signals require a display that is capable
of showing the individual fields in a sequential order.
And only CRT displays and some plasma displays are capable of doing so.
So, because CRTs are not in existence anymore, most displays are progressive
ones, interlaced video becomes less and less important as time goes on.
So in the special representation we see here the two
fields that I mentioned the top field and the bottom field.
Over here we see the progressive scans and in the time domain here is the frame rate,
so for NTSC this is one 30th of a second, the distance between two frames.
Why when we deal with interlaced video the distance
between fields now is one 60th of a second.
When we deal with I frames we need to take
the DCT of eight by eight blocks of intensity value.
So given a 16 by 16 interlaced macroblock we have
two ways to break it into four, eight by eight blocks.
The first one shown here is the field mode.
[UNKNOWN] the 16 by 16 macroblock is broke into
these four eight by eight blocks, where here we
put the pixels that belong to field one, which
is also called the odd field or the top field.
While in these two blocks we put the pixels that belong
to the second field or the even field or the bottom field.
Clearly if the frame mode is chosen then this 16 by 16 interlace macroblock
is broken into four, eight by eight, also interlaced blocks as shown here.
Clearly this is a rate control decision but needs to be made.
MPEG-2 supports both the frame and the
field prediction mode for motion estimation compensation.
So, according to the frame prediction, the two fields are treated as a frame.
It's an interlaced frame and the motion
estimation compensation is done at the frame level.
In the field prediction mode we have an example here
where this field could be a top field or a
bottom field, and can be predicted from either the top
field or the bottom field of the previous frame in time.
While in this example, we're looking at the bottom field of
this particular frame at time t, let's say, which can be predicted
from the first field of the same frame, the same time instance,
or the bottom field of the previous field, times t minus one.
So in this case, the distance here is one 30th of a
second, while the distance between these two fields is one60th of a second.
Another motion compensation mode provided in MPEG-2 is the dual prime mode.
This mode is applicable only to P pictures and to GOPs that
have no B pictures between these P pictures and the reference frames.
So for frame pictures, two motion vectors are associated with each field.
So for this top field here, the motion vector v1 and
pv1 are associated with this, this dotted line.
If we, if we denote by pv1 X, the horizontal component
of this vector, then pv1 X is encoded in terms of v1x.
And, as you can see here, V1x connects this
top fields which are one 30th of a second apart.
While pv1 corrects the bottom field of this frame and the top
field of this one, which are one 60th of a frame apart.
Therefore, pv1 is half v1x, shown here.
And delta of x is a correction there.
If you can see there, the vertical component of pv1.
It's the same course and duration.
It's half of the v1 of y.
Plus, minus this term here, which corrects, takes into account
the shift between a top field and the bottom field.
An even field and an odd field.
And here is the correction term.
So, what's included in the bitstream is not
pv1 and pv2, but instead this correction factor delta.
Similar considerations are made here, when we have the, the field picture.
This table summarizes the key characteristics of the profiles and levels
defined for both scalable and non-scalable MPEG-2 to bitstreams.
In this table the upper bounds for the
picture resolution, frame rate and bitrates are provided.
The bitrate data refers to the maximum compressed
bitrate supported by the input buffers of the decoder.
While NA indicates that, that there
is no conformance restrictions for these variables.
It's expected that any implementations of the standard will
support only the main profiles at the main level.
The main profile provides the basic functionality of MPEG-2.
The simple profile is a low-cost version of the main profile.
Coding is similar, as in the main, but without any bidirectional prediction.
Levels refer to picture resolution.
The upper bound of the main level is C, CCIR, 601 picture format.
The high 1440 and the high level
resolutions correspond to picture resolutions for HDTV.
The low level resolution corresponds to SIF format.
So, these profiles and levels have a hierarchical relationship.
That is, a syntax supported by higher profile level
includes all of the elements of lower profile levels.
A decoder that can decode bitstreams at a certain profile level should also
be able to decode bitstreams at the lower profile level than it's own.
We show here some of the systems supporting HDTV around the world.
So in North America the result of the grand alliance goes
to adopt for video encoding MPEG-2 main profile at high level.
This is the aspect ratio, the similar one with movies the
special resolution and then for audio, Dolby Advanced Coding three is used.
In Europe there is digital video broadcast.
It also adopted MPEG-2, but the simple profile at high 1440.
The aspect ratio is different.
The resolution is different, and for audio encoding MPEG layer two is chosen.
And finally in Japan, there is Muse, is
using the same aspect ratio as North America, the
resolution is slightly different, but the video is
also similar to main profile at high level MPEG-2.
Ознакомьтесь с нашим каталогом
Присоединяйтесь бесплатно и получайте персонализированные рекомендации, обновления и предложения.
Coursera делает лучшее в мире образование доступным каждому, предлагая онлайн-курсы от ведущих университетов и организаций.
© Coursera Inc., 2018 Все права защищены.
