Hi, my name is Brian Caffo. I'm in the Department of Biostatistics at The Johns Hopkins Bloomberg School of Public Health, and this is Mathematical Biostatistics Bootcamp--lecture eight, on asymptotics. In this lecture, we're going to take a trip to Asymptopia. Asymptopia is a land where we have an infinite amount of data, so it should be fun. We're going to talk about limits, but limits of random variables. And so there's intricacies that you have to account for when you consider random variables instead of standard mathematical limits. And it's a quite difficult subject, but we're going to show that we have basically two tools, the Law of Large Numbers and the Central Limit Theorem, that are going to be our primary methods for looking at random variables. So let me just review numerical limits first. And I'm not gonna go into too much detail, and I'm gonna kind of treat it here as heuristically. Just to illustrate, in case you asymptotic's a little bit rusty. Suppose I had a sequence, where A1 was .9, A2 was .99, and A3 was.999. And I, and I hope you can get the pattern at this point. So clearly, this sequence in some sense of the word converges to one. It converges to .9999999, that gets closer and closer to one. As, the element of the sequence, gets larger and larger. Well we can formalize this. The formal definition of a limit is for any fixed distance we can find a point in the sequence. So if the sequence is closer to the limit than the distance from that point on. So take, for example. This sequence. The distance between AN and one is ten to the - N where N is the point in the sequence. And so, if we pick any distance, say, epsilon, we can find an n so that ten to the minus n is smaller than epsilon. And then, because ten to the minus n just keeps getting smaller as n get larger, then it's smaller than epsilon from that point onward. So that satisfies our definition of limit. So clearly, this converges to one. And it's kind of an interesting fact that and infinite sequence of .9s and one is the same number. But that's kind of an interesting fact that if you ever take a class on real analysis though, they'll discuss that sort of thing at length. But anyway I hope you get this basic sense of a limit as'n' goes to infinity the sequence converges it looks more and more like it's limit so that the distance just gets closer and closer and closer and never gets bigger again. The problem with that is that, that only works for just a series of numbers, right? Now we wanna talk about say, limits of averages of coin flips. And then it gets much harder. So take, for example, the average. And now we're gonna talk about. In average comprised of n observations. So let's say x n bar and now instead of saying x bar like we typically do which is our average, we're going to annotate it by a subscript n to show that it's an average of the first n of a collection of IID observations. So for example x bar could be the average of the result of n coin flips which is the sample proportion of heads. Well, there is a limit theorem for averages, and we would say that xn bar converges in probability to a limit. And we, relate this back to the ordinary definition of limit, by saying, well, the probability that x is closer, than any specific distance converges to one. So in this case, probability xn bar minus the limit being less than, any quantity epsilon that you fix. Then that probability is a number pn. Right? And the definition of convergence in probability is that sequence of numbers, pn, converges to one, right. So we've converted the problem of what does it mean for a random variable to converge, we've converted that back to the definition of convergence of a series of numbers. We've said convergence and probability. Implies convergence of the collection of probability numbers in the standard sense of the definition of convergence. So now we have a way of talking about how random variables converge. So, establishing that a random sequence of variables actually converges is hard, as you can imagine, it's hard. If you look back at the previous definition, it's not the easiest thing in the world to think about. So we have something that makes it a lot easier for us, and that's called the law of large numbers. So, if you've heard people talk about the law of averages, typically they don't know what they're talking about. But probably they are referring to the law of large numbers. And the law of averages and there is the law of large numbers. Basically says that if x and x on a IID from a population from mean-mu and various sigma squares so in this case we are going to assume that the random variables have a variance. Its interesting to note that there are distributions where there is no variance. You try to calculate the variance and you get infinity or something like that. In this case we're gonna assume that there is a variance and it's finite. Then the sample average of IID observations always converges in probability to mew. And that's the, called the law of large numbers. Probabilists make a lot of distinction over various kinds of the laws of large numbers. And they've worked very hard to get kind of minimal assumptions for the law of large numbers to work. And in fact, we're using a very lazy version of the law of large numbers. They would probably upset at us for teaching this one, but it's okay, we don't care. The basic idea I want you to get is that, averages converge to mu. Averages of IID Observations converge to mu, the population mean, that, from which the observations were drawn. This is a good thing, right? This basically says, if we go to the trouble of collecting an infinite amount of data. Then we get the number that we wanna estimate mu, exactly. Right? Which is good, because collecting an infinite amount of data takes a lot of time. Actually, infinite amount of time. If you're willing to make this many assumptions, the Xs all have a finite variance. It's pretty easy to prove the law of large numbers using Chebyshev's inequality which, Chebyshev's inequality if you remember had a pretty simple little proof so this kind of very complicated idea, it's amazing that it has a fairly simple little proof. So remember that Chebyshev's inequality states that the probability that a random variable is more than k standard deviations from the mean is less than one over k squared. So therefore the probability that Xn bar minus mu, in absolute value, is bigger than or equal to k standard deviation of x bar sub n is less than or equal to one over k squared. Now, let's pick an epsilon. Pick any distance epsilon. Because remember to establish the convergence of a limit of numbers we have to pick an epsilon. And now to establish convergence and probability, we have to show that the probability that xN bar minus mu, being bigger than epsilon, goes to zero or being less than epsilon goes to one, those two statements are equivalent. And so let's let K, from our previous definition, be epsilon divided by the standard deviation of Xn bar. K is not a random variable, right? Epsilon is a number that we pick. And standard deviation of Xn bar, Xn bar is a random variable, but the standard deviation of it is sigma over square root N. Okay? So this is just a number, there's nothing random in our definition of K. So if you plug that back in for K, right? You get the probability that Xn bar, minus mu, being bigger than epsilon, is less than or equal to, the standard deviation of Xn bar squared, which is the variance, of Xn bar, divided by epsilon squared. And we, from a previous lecture, already calculated what the variance of Xn bar is. It's sigma squared over n. So this probability is, less than or equal to, sigma squared over n, epsilon squared. Now as n goes to infinity, sigma squared isn't changing and epsilon squared isn't changing and the n's in the denominator, so this whole thing goes to zero, okay? So the probability that the random average xm bar is more than epsilon away from the mean goes to zero as n goes to infinity or we stated the probability that xm bar is less than. Epsilon from the mean goes to one as N goes to infinity. So either of those statements equivalently say that Xn bar converges in probability to mu. I think it's kind of staggering that it's really basically two lines is all you need to establish this fairly complicated result. Now on the next, page, I just have a, simple example where I, simulated random normal with a mean of zero, and I show the cumulative sample mean. So I took one random normal, and then I took that random normal and generated a second random normal and then averaged it. And then generated a third random normal and just averaged it with a remainder. And the iteration at the bottom is the number of observations that goes into that mean, right? And then on the vertical axis it shows the value of the average. And you can see if, first, there's quite a bit of variability, right? Remember, the variance of the average is sigma squared over n. The variability is going to zero and this dash line is the asymptote. Right? And as you can see this average, as we include more observations in it, is going to converge to this. You can see it already converging a little bit by 100 iterations to this dashed line. That's simply the law of large numbers. So let's cover some useful facts about the law of large numbers. One Interesting fact is. Functions of convergent random sequences converge to the function evaluated at the limit. So. This includes sums, products, differences. So, for example Xn bar, squared, converges to mu squared, right? Because, x bar converges to mu, and so this is just a function of x bar. So Xn bar squared converges to mu squared. Something different is that average of the squared observations converges to a different entity okay, right? Let's go through this a little bit carefully, just because it's kind of an odd little point. So Xn barred squared converges to mu squared, but if we sum up Xi squared the individual observation squared and divided by n that no longer converges to mu squared. Well why not? Well it's the difference between the square of the average and the average of the squares. So, in this case it's the average of the squares. In this case each xi squared is a random variable so we could just call it y instead of Xi squared. And then their average of these Xi squareds or average of these Ys is gonna converge to the population mean of those Ys. Well we can calculate that. We know what the expected value of Xi squared is because we can use the shortcut formula for the variance. Which, recall, was expected value of Xi squared minus expected value of Xi quantity squared. We can just work that formula to solve for expected value of Xi squared. And show that that's equal to sigma squared plus mu squared. So, the average of the squared observations converges to sigma squared plus mu squared, whereas the square of the average converges to mu squared. So it's kind of an interesting little point, but just remember that those things are different. And by the way, this little fact that we just sh-, showed, we can use this to prove that the sample variance converges to sigma squared, and we'll do that on the next slide. So let's actually go through this proof that the sample average converges. To sigma squared. And I think you'll see in the process of the proof that it doesn't matter whether we divide by n or n - one, it's going to converge to sigma squared. So here we have the definition of the sample average, summation xi - xmbar squared, all divided by n - one, we're going to use the unbiased estimate of the sample variance. Well, recall there was a shortcut formula for the sample variance and it worked out to then be the numerator had a shortcut formula that was summation Xi squared - n xbar squared. And so we're going to use that formula. And then we get summation Xi squared over N minus one minus N barred squared over N minus one. And let's just rearrange terms, and multiply and divide by some Ns because that N minus one is a little annoying. So we have n over n - one times summation xi squared over n - n over n - one Xn bar square. Let's look at each of these things in turn and remember, from the previous slide I told you, I didn't prove this, and just have to take it as true, is that, you know, if you multiply convergent sequences they converge to the product of the limits. If you add and subtract convergent sequences they converge to the difference of the limits and so on. So let's look at each of these terms one at a time. N over n minus one, clearly that converges to one. You don't believe me, plug n over n minus 1n for very big value of n in your computer and you'll see that it gets closer and closer to one. Probably the easiest way to see this is it's one over, one minus one over n and that one over n clearly goes to zero. Okay. We just on the previous slide talked about how, summation Xi squared over n converges to sigma squared plus mu squared. So, the second term converges to sigma squared plus mu squared. And we have minus n over n minus one, which again converges to one. And then Xn bar squared converges to mu squared. We talked about that on the previous slide. So we have this expression right here, sigma squared plus mu squared minus mu squared, which is just sigma squared. So that proves that the sample variance converges to sigma squared, and then of course, the sample variance if we happen to divide by the bias sample variance, if we happen to divide by N instead of N minus one, also converges to sigma squared. And then we can square root the sample variance and get the sample standard deviation and see that it converges to sigma as well, just by the rule from the previous page where we said that functions, in this case the square root function, of convergent random variables converge to the function of the limit. So what we've found is, we have our law of large numbers, and with a couple of rules that we just stipulated, we've got that the sample mean of IID random variables converges to the population mean that it's trying to estimate. The sample variance converges to the population variance that it's trying to estimate. The sample standard deviation converges to the population standard deviation that it's trying to estimate. And in all these cases you see the pattern that the sample entity converges to the population quantity that it's trying to estimate. Basically saying, that if you go to the trouble of collecting an infinite amount of data. Then you actually get the value you that want to estimate. You don't get it with noise, you get the actual value. We give this name a property and we say that an estimator is consistent if it converges to what you want to estimate. And the Law of Large Numbers is basically saying that the sample mean is consistent, and then we know, now know that the sample variance and the sample standard deviation are consistent, and it doesn't matter whether you're dividing by n or n - one, they're all consistent. But also remember the sample mean and the sample variance are unbiased as well. And by the way the sample standard deviation is not unbiased. Consistency by the way is a very weak property so the sample standard deviation is biased, unlike the sample mean and the sample variance. Consistency is a very weak property. Saying that an estimator is consistent is not even really a necessary property. It, it seems like it should be necessary, but if, if something converges to mu plus epsilon where epsilon is a miniscule number that is of no importance, then that estimator is not consistent. So it's fair to say that consistency is sort of a, kind of a weakly necessary but definitely not sufficient property for an estimator to be useful. We have also seen that being unbiased is neither necessary nor sufficient for an estimator to be useful either. For example, we've talked about the biased variance trade off that estimators can be slightly biased and you can want that, because you improve on the variance. So, what we're winding up with is a collection of properties that describe estimators. And you really need to think about the collection of properties as a whole to evaluate an estimator. And these various mathematical concepts are, are useful but they never, in isolation, tell the full story on the utility of a estimator. They might be useful for eliminating really dumb things, if something's definitely not consistent in a way that it doesn't converge anywhere near the estimate. Probably that's not something that you wanna use. But apart from those kind of stark circumstances, these properties you need to take as a collection to try and decide which estimators are the right ones to use. So let me give an example of an estimator that's consistent but not very good. So take the data, and, only take the first half of the collected observation. So we have, instead of Xn bar, we have Xn over two bar, right? That estimator is, of course, consistent, as n goes to infinity. You know, it just has, fewer observations than if you took all of them. But still the n is going to infinity in this case. It's just n over two. So that estimator's consistent, but it's got an obvious better estimator right in front of you. Basically, the estimator using all of the data. So there's a, a particular example of an estimator that's not consistent where there's a better estimator that comes to mind. Here you have to actually account for the fact that the estimate with all of the data has a lower variance than the estimate with half of the data. So that's enough discussion of limits and the law of large numbers. Next we're gonna go on to the central limit theorem. A very important theorem in statistics.
