0:00

Welcome back to the course on audio signal processing for music applications.

Â In the previous theory lecture, we started talking about sound transformations.

Â In particular, about how to use the short time free transform and

Â the model to manipulate sounds.

Â In this second theory lecture, we want to continue that,

Â and talk about the harmonic plus residual model and

Â how it can be used for transforming the frequencies of the harmonics.

Â And then, the harmonic plus stochastic model, and

Â how can it be used for time stretching and morphing operations.

Â 0:41

These morphing by the way is quite different from the one that

Â was possible with the short time period transform.

Â And of course, these are just some example operations, example transformations.

Â But the number of possible uses of these models is much larger.

Â 1:01

Let's start with the harmonic plus residual model.

Â This is the plot diagram that we have seen before in which from the input sound,

Â we analyze the spectrum, find the peaks.

Â And then, find the fundamental frequency, find the harmonics and

Â we subtract those harmonics from the original signal.

Â We need to compute the spectrum of the original signal again in order

Â to develop a spectrum from which we can subtract these harmonics in a proper way.

Â 1:34

Okay, and then where we do the transformations is in the frequencies and

Â amplitudes of the harmonics.

Â We don't apply any transformation to the residual in order to

Â apply transformation to the residual it make sense to have a model for it.

Â And that what we are going to be introducing

Â in the next model the Harmonic plus sarcastic model.

Â 1:58

And here, the transformations on the harmonics, again, like in the sinus solar

Â model, it's applied only on the frequency and amplitudes.

Â The phase are forgotten, we don't use them,

Â we will be re-synthesizing them again after transformation.

Â 2:15

And the main issue of that is because that phase is a very sensitive to

Â transformations and it's very difficult to handle them.

Â So it's better to just regenerate them after transformations and

Â it doesn't sound that bad.

Â Anyway, so after the transformations we can

Â sum these new harmonics to the residual signal and

Â these residual signals since it does not include any deterministic.

Â Any harmonic information, it will merge quite nicely.

Â And then, it will create a sound that has the same residual as the original and

Â the harmonics will be modified.

Â 2:53

In terms of types of transformation that can be done to these harmonics

Â to the frequency of these harmonics.

Â There are many and here I explained three common ones.

Â The most traditional one is to just transpose

Â all the frequencies by a given factor.

Â So, we multiply all the harmonics by a scaling frequency factors so,

Â that would correspond to a transposition so we multiplied by two.

Â It would correspond to converting the sound to an octave higher if

Â we multiplied by 0.5 it would be an octave lower.

Â Then instead of multiplying, we can just shift all the frequencies.

Â For example, we can sum a factor to all the frequencies, so

Â we'll be shifting all the harmonics by an additive component.

Â 3:47

And finally,

Â the last one I want to mention is what is called frequency stretching.

Â In which, we are applying frequency change that is dependent on the harmonic.

Â So here, what we are doing is basically dividing every harmonic by it's harmony

Â number, so basically get a frequency around the fundamental.

Â And then, we multiple by the harmony number to the power of a scaling factor.

Â Therefore, the higher harmonics will be modified very

Â differently from the lower harmonics, it's very much harmonic dependent and

Â that can create some nice effects that we will see.

Â 4:30

So this is a plot of these three transformations that

Â exemplify these operations quite nicely.

Â So on the top left, we have the original harmonics all located

Â at the harmonic number location, one two, three, four, and five.

Â Then if we transpose by two on the right,

Â it's just simply every by two of harmonic one in position two,

Â harmonic two in position four, etc.

Â We are maintaining a harmonic spectrum with just transposing by a factor or two.

Â If we shift instead by a given value, in this case since

Â we are having the frequencies normalize to integer values.

Â Here by 0.5 will be shifting by half of the frequency

Â of the first fundamental, of the first harmonic.

Â And in here, we see that we're shifting everything by a constant value.

Â From now after this transformation, we do not have any more harmonic series

Â because the first harmonic is not just the fundamental frequency anymore.

Â Okay, so we have now a series of values that are equally spaced but

Â they are multiples of another frequency.

Â So the sound may be quite a bit different and quite interesting in some cases.

Â And finally, the frequency stretching operation,

Â it's a quite different operation.

Â Now it changes the distance between the harmonics and

Â it is like an accordion.

Â So if it is here at one point three means that well the first

Â value the fundamental will not be touched.

Â Then first one will be stretched by a factor of 1.3 and

Â then these are going to be powers of these.

Â So as we go higher up, it's going to go sort of the distance we'll increase

Â kind of exponentially and these create again quite very interesting effects.

Â 6:44

So let's listen to sound in which we have applied some stretch and

Â transpose kind of transformation.

Â So let's listen, so it's a flute sound that we can listen to.

Â [SOUND] Okay, A4 below that we see the harmonics

Â that have been identified on top of the residual spectrum of the original signal.

Â And then, what we're doing is leaving the residual cities so

Â that the background spectrogram is exactly the same as the top one.

Â And we're just modifying the harmonics, so

Â if you look here, the harmonics have been heavily modified.

Â And if you see they have been stretched and

Â at the same time transposed a little bit, so let's listen to that modified sound.

Â [SOUND] Okay, of course quite different.

Â And if you pay attention the residual is there and since the residual is basically

Â these breath noise is emerges also quite well with these transpose harmonics.

Â 8:02

It's very similar to the previous one and now, the transformation is applied both

Â in the modeling of the receiver signal and of course in the harmonic component.

Â So we have an stochastic approximation and therefore,

Â we can transform that quite easily because it's a model that has simplified

Â that the residual quite a bit and is quite flexible in terms of manipulations.

Â 8:47

And so in terms of what is actually done is very similar to what

Â we presented to the sinusoidal model.

Â Now, what we are introducing is the transformations

Â on the stochastic component in a similar way.

Â So the frequencies of the harmonics can be scaled by a factor and

Â the time of the reading of these frequency values

Â also can be changed in order to obtain these time scaling operation.

Â So we have this scaling function, the amplitude is the same way and

Â the stochastic component is handled in the same way, okay?

Â And of course, the phases are generated and

Â they are generated by starting from an initial phase.

Â And then adding the frequency of every harmonic that we want

Â to generate to that phase, so we can generate the instantaneous phase that way.

Â 9:46

Okay, let's show an example of that.

Â This is the sax phrase that we are also we have heard before.

Â Let's listen to that again.

Â [SOUND] Okay, and then we have the analysis of it.

Â Where we have analyzed the harmonics and

Â the stochastic component and then we can change that.

Â In this particular case, we have done time scaling, so we have changed the timing and

Â now with the residual approximating residual stochastic,

Â it's very easy to do time scaling of the stochastic and the harmonics.

Â And that's it,

Â we have just focused on the timescaling transformation in this particular example.

Â Let's listen to that, a very simple timescaling operation.

Â [SOUND] So, basically the only thing we have done is we

Â have compress the first part of the signal by quite a bit.

Â In fact, it's like half of the duration and

Â the second half has been extended by twice as much.

Â So in fact, the overall duration of the sound remains the same and

Â of course as you can imagine, this is very flexible and

Â we can do a lot of envelopes that we can play around with.

Â Okay, now, let's talk about the morphing using this harmonic plus stochastic model.

Â Okay, here we have simplified the block diagram.

Â We have two sounds, x1 and x2.

Â Now they're basically at the same level and basically,

Â what we're going to do is interpolate the two representations.

Â So from x1, we obtain the frequencies and amplitudes of

Â the harmonics and the stochastic approximation of the residual.

Â And of sound two we do the same thing and then,

Â what we are doing is interpolating these two sets of functions.

Â We are interpolating the frequencies, the magnitude of the harmonics and

Â we are interpolating the stochastic envelopes of the residual.

Â And then of course, we can synthesize back the output sound by

Â generating the stochastic component and the sinusoidal component.

Â 12:11

Let's look at the particular frame of a sound in which we have applied like

Â a 50% interpolation, so on top left is X1.

Â One particular frame the harmonics

Â visualize as the location is the frequency and

Â the height is the amplitude and then the blue one is sound one.

Â X to the red one is a sound too and then, we can interpolate these two.

Â Basically, harmonic by harmonic we can interpolate and as you see,

Â the X2, it's a different frequency.

Â It's a sound that has a higher pitch, so

Â the interpolated result has a pit which is in between the two,

Â and has a shape again that is in between the two, okay.

Â So it's basically, a 50% interpolation between the two set of values and

Â the suppressing component below does the same idea.

Â So on X1, this shape is the approximation of the residual,

Â so the [INAUDIBLE] is where we actually have values has an envelope.

Â And then X2 is another envelope for X2 and the output is the interpolated envelope,

Â so it's a shape that is in between these two shapes.

Â 13:55

So here, we see on the top the violin sound, and we can listen to.

Â [SOUND] Okay, and it's analyzed, so we see the harmonics.

Â Well, we see I think here is the first like 40 harmonics and then,

Â the stochastic component of that.

Â And then, we see below the analysis of the soprano sound,

Â let's listen to this [SOUND] Okay, where we see again the harmonics,

Â it's a higher pitch and the stochastic component in the background.

Â And now,

Â we can choose the interpolation values to go from one set of values to the other.

Â So it will progressively go from one sound to the other.

Â Let's listen to the result.

Â [SOUND] Of course, since the soprano sound is

Â higher than the violin, we hear this glissandi.

Â But at the same time, we hear the evolution of the timbre, and

Â every single parameter is slowly changing from one to another.

Â Again, the possible type of Functions to control these interpolations

Â is enormous so we can play around quite a bit with these ideas.

Â 15:13

That's all, so again, Wikipedia, there is not much and there is not much that we can

Â refer to in terms of trying to understand some of these transformation aspects.

Â And in fact a lot of these things is just by trying, so

Â I would encourage you to try these things and

Â develop your own intuition of what it works and what it doesn't.

Â And that's all, so

Â this has been the second theory lecture on sound transformations.

Â Of course, in this week, what is important is the programming and

Â the demonstration classes, even more than the theory lectures.

Â Because after all, it's a very applied topic, and it has to be grasped

Â from an application, and from an intuition and musical perspective.

Â So thank you very much for your attention and I see you in the next lecture.

Â Bye-bye.

Â