>> In this video, I'm going to talk about how we combine what we see and what we hear to synthesize a combine percept of visual and auditory information. Our brains are constantly combining what we see and what we hear, but we're usually not aware of it because the process operates so seamlessly. Once again, illusions are a great way to get a sense of what's going on under the hood. By fooling ourselves, we can see what our brains are really trying to accomplish. So, I'm going to show you an illusion called the McGurk Effect. It involves the integration of what we see and what we hear in order to understand speech. I'm going to play a video and then I'm going to use the end video quiz questions to ask you what you perceive. So watch this person speak, and try to figure out what nonsense syllable he's saying. >> Da da, da da, da da. >> Well if you thought he was saying da da or tha tha or la la you actually were incorrect. I'm going to play it for you again. This time close your eyes and don't watch his mouth as he is speaking. >> Bah bah, bah bah, bah bah. >> If you listen carefully, the sound track is bah bah, but the visual portion is of this person saying, gaga with a g sound. Now making the sound b and making the sound g involve very different lip motions and your brain puts together the visual information that says that it couldn't have been a b sound because this gentleman's lips did not come together and close at the front of his mouth when he was speaking. Since the visual system is saying it couldn't have been a bah sound, the brain synthesizes a compromise interpretation, that, that is usually da da, tha tha or la la. It's a lot like a Pinocchio effect that we talked about earlier, where the brain had two competing pieces of information and it had to up with a, a compromise interpretation that fit both of these pieces of evidence. Now one of the things that I think is really very, very interesting about this particular illusion is that even when you know what's going on you still perceive the illusion. So I'm going to play it one last time and my guess is that if you experienced the illusion the first time, you will probably still experience the illusion even now that you know what's going on. >> Da da, da da, da da. >> I generally hear this illusion every time I play it. Well, another form of visual and auditory interaction involves the ventriloquism effect that we've talked about previously. And these two illusions are closely related to each other. They both involve speech and they probably both involve spatial location as well. Because it is by knowing that a visual stimulus and an auditory stimulus are coming from the same location and space, that you decide that they must have arisen from a common underlying event. Those are the visual and auditory components of our rich visual and auditory environments that we want to combine. But combining visual and auditory information in space is made very challenging by the fact that different reference frames are used for each of these systems. You already know from previous videos that the way that we know where a visual stimulus is located is by knowing where on the retina the image of that stimulus falls. This is an eye-centered frame of reference because it matters where the stimulus is with respect to the direction that they eyes are looking. When the eyes are looking in this direction, the image of this bird will land right there on the retina. When the eyes are looking in this direction, the image of the bird will land at that location on the retina. In contrast, for the auditory system, the clues that the brain assembles to figure out where a sound is located generally depend on the location of the sound with respect to the head and sometimes to the ears. Interaural timing differences vary with the location of the sound with respect to the axis of the ears. Interaural level differences depend on the shadow cast by the head and thus also depend on the axis of the head. Spectral cues do depend on the position of the ears and if you have mobile ears that's going to matter, but in humans the ears are immobile and thus the ear-centered spectral cue information is essentially the same as head-centered cues. These frames or reference refer to the raw information that the ear is providing to the brain. They do not refer to the way that the brain actually encodes that information because as we'll see the br, that's a little more complicated and the brain is able to transform signals out of their native reference frame into something that's more appropriate for communicating with other sensory systems. So, visual information is eye-centered and the auditory cues upon which the brain deduces sound location are generally head-centered. So, what is the relationship between these two reference frames? If your eyes are directed straight ahead, then the visual object that is making the sound right here will be located to this side of your visual reference frame. It will also be located to the same side of your auditory reference frame, but if you move your eyes over to this location, now the location of the visual image has switched on the retina, but its location with respect to the head and the ears will be unchanged. So the same location with respect to the head and the ears can occupy very different locations on the retina depending on the direction the eyes are looking with respect to the head. What this means is that it's going to be very tricky to reconcile what we see and what we hear. If you have a group of auditory sources in the world, there is a potential correspondence between every point in visual space and every point in auditory space. Furthermore, as the eyes move around, those correspondences are going to shift. So the true correspondence is governed by eye position relative to the head. That means the communication between the visual and auditory systems is critically dependent on information about eye position. And in a previous video we talked about how exactly the brain knows about eye position and the importance of retaining a copy of the motor command to move the eyes in order to know where our eyes are directed in space. Building on that, my laboratory and other laboratories have found that eye position modulates activity at multiple levels of the auditory pathway. In fact, we have found effects of eye position on auditory processing in all the places that we've looked, and this suggests that there are widespread interactions between the visual and the auditory pathways. Let me unpack this a little bit and tell you about the auditory pathway, what some of these brain regions are and how they're organized in sequence. So, this is the ear and here is the cochlea and signals from the cochlea travel into the brain via the auditory nerve. They pass through several stages before reaching auditory cortex, which is located generally in this region. It's important to note that, in the auditory pathway in particular, signals travel in both directions including back out to the ear itself. So, information is flowing in both directions along the following chain. Starting with the cochlea in the ear and traveling to the cochlear nucleus in the brain and from there to the superior olivary complex to the, a structure called the inferior colliculus, which is next to the superior colliculus, and you've heard about the inferior colliculus in an earlier video, to a nucleus of the thalamus called the medial geniculate nucleus and finally to auditory cortex. And there are actually multiple stages after that point, but this is certainly plenty to think about for right now. So we and others have found that in auditory cortex neurons are sensitive not only to sounds, but also to the orientation of the eyes as the sounds are being presented, and we have found similar results in the inferior colliculus as well. And we simply don't know if a similar effect might be present in these earlier stages of the auditory pathway. Illustrating this schematically what we know is this, when the eyes are directed to one position and a particular sound is presented, a particular neuron might respond like so. If the same sound is presented, but the eyes are moved to some other location, the response of a given neuron might vary a little bit, and again, play the same sound but move the eyes in the opposite direction, you get a slightly different response. So the auditory stimulus is not varying in these three conditions, Only the eye position is varying, and yet the response level is varying as well. Behavioral evidence that this kind of eye position effect is relevant for the integration of visual and auditory space comes from experiments involving the ventriloquism aftereffect. Together with Norbert Kopco and Barbara Shinn-Cunningham we have investigated how the ventriloquism aftereffect depends on eye position. And the experiment is something like this, we present a fixation target at a particular location, subjects move their eyes to look at that target. We then present a paired visual and auditory stimulus. We don't actually present ventriloquists, we use simpler laboratory stimuli like noise bursts and little LED lights, but the principle is the same. And the idea is that after being exposed to a visual stimulus at one location and a sound coming from a different location, subjects will perceive the sound is coming from the location of the visual stimulus that is capturing that perceived sound. The aftereffect comes in when the sound is presented alone and subjects make an eye movement to the location where the visual stimulus had been. That shift in the eye movement is what we call the ventriloquism aftereffect. And what we found is that the ventriloquism aftereffect was biggest when the sound location and eye position matched the original exposure. So, for example, if this was our original fixation position, we got the biggest effect when we presented the sound at the same location that had been used for the visual and auditory pairing. If we moved the sound over to that location, we got a smaller ventriloquism aftereffect. And if we move the eye position over, we got a small aftereffect at that position. And we got a small aftereffect at the original location. This suggests that both the head and the eye-centered location of the sound is important for governing the size of the aftereffect and it is only when both head and eye-centered locations both correspond to the original exposure that the maximum learning occurs. Here are some of the references from my group and from other groups concerning these effects of eye position in the auditory pathway as well as the effects of reference frame on the ventriloquism aftereffect. So far what we've talked about is eye position modulating activity in the auditory pathway and modulating the behavioral impact of a visual stimulus on the perceived location of a sound. We actually have evidence that the brain goes much farther than that and actually translates auditory information into the same frame of reference used by the visual system, that is, into an eye-centered frame of reference and I'll tell you about that in the next video.