Hi, my name is Brian Caffo and this is Mathematical Biostatistics Boot Camp Lecture twelve on Bootstrapping. Today, we're going to talk about the tool of bootstrapping which is an incredibly useful, handy result in statistics that you can use in a variety of settings. It was made, sort of coincided with the personal computer revolution. And so, it, it gives us a way to avoid an awful lot of mathematics in biostatistics. Before we talk about the bootstrap, we're going to talk about the jackknife, which is a precursor to the bootstrap that is as its name suggests, a handy little tool. So, let's talk about the jackknife here briefly, before we go into the bootstrap. The jackknife is as exactly as it's name suggests, a handy little tool. The bootstrap, on the other hand, is like an entire workshop of tools. The key idea in both the jackknife and the bootstrap is, to use the data, so called resampling of the data, to get at quantities that are difficult to get at otherwise. For example, variances, and biases, and that sort of thing. Now, we, we don't need either the bootstrap or the jackknife for something like the sample mean where we know all its theoretical properties. But, for other less obvious statistics, we need something that does it for us and, You know, it'd be preferable if that something didn't require a year of mathematics just to get us to the starting point. And, in contrast, the bootstrap is you dream up a, statistic or something like that. And you want to estimate a standard error with it, you can start bootstrapping it immediately. So, Let's talk a little bit about what the jackknife does before we begin with the bootstrap because sort of, historically, the jackknife came first. The first use of the jackknife was by the statistician, I mean. A butcher. His name, but I think it's pronounced Quenouille. And, he used the jackknife to estimate bias, I believe. Then, the jackknife was really popularized and further refined by the extremely well-known statistician, John Tukey, who we talked about a little bit in the lecture on plotting. Tukey had numerous inventions including the fast discreet Fourier transform. He coined the term bit for binary digit. He was the first person to do that, And he did lots of things. He invented the box plot. I think when you see it, you'll conclude along with me, the jackknifes a handy and incredibly clever thing for someone to think of. So, the idea behind the jackknife and similar to the idea behind the bootstrap is to, you have something you don't know, like the bias in a statistics, or the standard error of a statistic, and the idea is to use the data to get a sense of it. Wellm what the jackknofe does is it says, okay, well, one way to get at these quantities is to take one of the observations out. And then, formulate the statistic on the remainder and see how well the statistic does, you know, at estimating that one pulled out observation. And this is very related to the idea that, you know, frequently you hear of in machine learning and statistical prediction of so-called cross validation. The jackknife tends to have a different goal. In that, the goal of the jackknife tends to be bias estimation, or variation estimation. But the principle is very similar in that you're deleting observations. Leave one out cross validation is typically used as an estimate of prediction error. So, anyway, let's just focus on the jackknife. And if you take classes in machine learning or something like that, you'll talk about cross validation. The jackknife deletes one observation and calculates whatever estimate you're thinking of based on the remaining n - one of them. And then, it uses this estimate based on n - one of them in which you get n estimates having left out one observation one at a time. It uses these n estimates to do something like, estimate biases and standard error. And again, no, we don't need this for the sample mean. We know that the sample mean is unbiased under certain assumptions, and we know exactly what the standard error of the sample mean is under the standard setting. So, the jackknife isn't necessary for those settings, but it's, maybe necessary for other ones. So, let's just consider the jackknife for univariate data. And let's let x1 to xn be a collection of univariate data points where we want to estimate a parameter theta. And so, let's let theta be the estimate based on the full data set. And then, let's let theta hat sub i. Be the estimate of theta that you obtain, where you use the n - one observations obtain by deleting observation i.. And then, let's let thta bar be the average of the leave one out estimates. So, with that notation in mind, the jackknife estimate of the bias of our statistic theta hat, is just n - one theta bar minus theta hat. So, let's kind of consider the principal of this before we've, get to why in the world that n - one is there. So, theta hat is our estimate. Looking at how close it is to the averages of estimates where we deleted an observation each time, Is exactly going to give us a sense of kind of population level bias. And then, you might wonder, where in the world does this n - one come from? It's, factor that's based on the, For example, the sample variants where you would experiment the bias of the sample of variants, it would give you the correct answer. The n - one is sort of calibrated by, statistics that we actually know. So again, this estimate is really related to how far the average delete one estimate is from the actual estimate. And then, this n - one is just a factor that was sort of, a good estimate of what is the appropriate multiplier to have to get the bias to be an estimate of the true bias. And then, the jacknife estimate of the standard error is n - one over n times the sum of the squared deviations of the delete-one estimates around the average of the delete-one estimates. So, it's sort of like the square root of n - one times the variance of the delete-one out estimates. You, so, again, the rationale for this factor out front, The extra n - one, why not just, why not just take the variance of delete-one out estimates as an estimate of the standard error of the statistic? Well, it turns out that delete-one out estimates because they have the majority of the data. They have n - one of the data points included. They tend to be quite close to one another, And excessively close to one another. So, the variance, by itself, is not a good estimate of the standard error of the statistic. So, we need a fact, and they calibrated that n - one is a reasonable factor to do that, And the same thing is true with a bias. That, to delete-one out statistics tend to be a little too close to one another unless you sort of multiply this by its estimate by a little, but you don't get reasonable estimate. So, let's go through an example. So, we had 630 measurements of gray matter volume from workers from a lead manufacturing plant. The gray matter volume wound up to be about 589 cubic centimeters. And, we want to estimate the bias and the standard error of the median. And then, I'll come back to this discussion of jackknife the median because that's where we're going to move forward to the bootstrap. So, for example, the gist of the code to do this. Now, you don't actually have to execute the code. I'll show you in a page, how to do it. But, you can do it in any language, not just R. You just have to figure out how to delete observations one at a time. So, let's let n just be the number of observations we have. Theta hat is the median of these grey matter volumes. And then, the jackknife estimates are the median that I obtain each time where I delete the i-th observation, This sapply function is exactly that. Then, theta bar, just exactly from the notation from the previous couple of slides is just the mean of these delete-one out jackknife estimates. Then, my bias estimate is going to be n - one times the difference between theta bar and theta hat. And, the standard error is going to be the square root of n - one times the average squared deviation of the jackknife estimates around the average of the jackknife estimates. And then, on the next page, it's a lot easier to do this. [laugh] If you want to just use the software in the bootstrap library, you can jackknife, out is the jackknife function is the list of my grey matter volumes and the function I want to calculate the jackknife estimate of is the median. And then, I assign that to a variable out, then I pick out the standard error and the bias calculation. Both methods yield a estimated bias of zero and a standard error of 9.94. And, There's an odd little fact. The jackknife tends to work well for sort of smooth functions, and empirical quantiles often don't satisfy that requirement. The median is an example. So, it's an odd little fact the jackknife estimate of the bias for the median is always zero when the number of observations is even. So, the medians an example where the jackknife isn't that good of a thing to do. In general, if your function of the data, a nice smooth function, The estimate that you're getting is a nice smooth function of the data, then the jackknife will work fine. But, if it's not, then it tends to work pretty poorly. In that, there was a very well known paper by Efron, the inventor of the bootstrap that illustrated this quite starkly. And the jackknife has been shown to be a linear approximation of the bootstrap. So, if you're in some setting where it's going to be difficult to program off the bootstrap, then doing a jackknife, which is a pretty simple thing to do, is a handy little tool to use. And then, just to remind you, you know, don't use the jackknife for sample quantiles. It's a handy procedure and it works in a lot of settings, but maybe not for sample quantiles, like the median, as it's been shown to have some poor properties. And what could you possibly use then? Well, why not try to use the bootstrap. So, let's move on to the bootstrap which is maybe a little bit more of a complete toolbox but it's certainly a little less compact of a tool than the jackknife in exactly the way the analogy to the tools sounds like. By the way, the term bootstrap comes from this idea of pulling one out by ones own bootstraps, right? And, you know, of course, This has been discussed a lot. It's kind of an unfortunate title for a statistical procedure, because it makes it sound like the information's coming from nowhere, Right? Because you can't pull yourself up from your own bootstraps. It's physically impossible. But, you know, there's been plenty of theoretical work that shows where the information is coming from, from the bootstrap in, sort of, when it is applicable. Another thing I would note is this idea of pulling oneself up from one's own bootstrap is from the fable of Baron Munchhausen. And so, there's a great movie called The Adventures of Baron Munchausen. And it was done by some of the people who made the Monty Python series. If you get a chance, you should, you know, in honor of this lecture, watch the Baron Munchausen movie. But, at any rate, from that fable is where the term, pulling oneselves up from one's own bootstrap comes from. And then, that's where they got the idea for the name from this procedure. Any rate, Back to the jacknife. So, another way to think about the jackknife is this idea of so called, pseudo observations. So, if you take n times theta hat minus n minus one times theta hat sub i, you can kind of think of these as whatever observation I contribute to the estimate of theta. And then, notice that if, if the theta hat is the sample mean, then these pseudo observations are exactly the data themselves. So, it's sort of this idea of taking what worked in a very neat and tidy sense for the sample mean in trying to extend the idea to other statistics. And then, the sample standard error of these observations is the jackknife standard error. And, the mean of these observations is a sort of bias corrected estimate of the parameter that you're interested in. So, it takes your ordinary estimate and attempts to correct the bias. I have to admit, for my thinking about the jackknife, I kind of prefer to think about it this way in terms of the pseudo observations than in the, sort of, classical development of it.
