Loading...

Resampling Caveats

Course video 9 of 69

Learn the basics of statistical inference, comparing classical methods with resampling methods that allow you to use a simple program to make a rigorous statistical argument. Motivate your study with current topics at the foundations of science: publication bias and reproducibility.

Practical Predictive Analytics: Models and Methods

Вашингтонский университет

Курс 2 из 4 — Наука о больших данных Специализация

Statistical experiment design and analytics are at the heart of data science. In this course you will design statistical experiments and analyze the results using modern methods. You will also explore the common pitfalls in interpreting statistical arguments, especially those associated with big data. Collectively, this course will help you internalize a core set of practical and effective machine learning methods and concepts, and apply them to solve some real world problems. Learning Goals: After completing this course, you will be able to: 1. Design effective experiments and analyze the results 2. Use resampling methods to make clear and bulletproof statistical arguments without invoking esoteric notation 3. Explain and apply a core set of classification methods of increasing complexity (rules, trees, random forests), and associated optimization methods (gradient descent and variants) 4. Explain and apply a set of unsupervised learning concepts and methods 5. Describe the common idioms of large-scale graph analytics, including structural query, traversals and recursive queries, PageRank, and community detection

Просмотреть программу курса

Получаемые навыки

Random Forest, Predictive Analytics, Machine Learning, R Programming

Преподаватели

Bill Howe
Director of Research

Текст видео

So backing up a step, we talked about a shuffle test for significance.

And that had a resampling method, but

it was a little bit different because we shuffled the labels.

And then we talked about bootstrap samples to drive a confidence interval and

really any other kind of statistic you may need.

One question is, why do we even need that shuffle test at all?

Why can't we just use the confidence interval itself?

After all,

it's possible to reason about significance even with a confidence interval.

You can see whether zero falls within the confidence interval or not.

The reason is is that there's a sort of a subtle reason about what you're

starting from.

If you compute the confidence interval, you're already asserting that there is

a significant difference, say in the means, or the statistic is significant.

When you're testing significance,

you're starting with a null hypothesis and you're deciding whether to reject that or

not, and you can see intuitively when that might make a difference.

Consider a admittedly contrived case where you only have two data values,

one treatment and one placebo.

Sorry, I said placebo, in this case, well let's go with placebo.

So one is a treatment, the other's a placebo, and

we're trying to measure the difference between them, right?

And let's imagine this is still survival days, so with the treatment,

they survive long.

And there's one patient to survive longer, 36 days, and

with the placebo they only survived 27 days.

So a 95% confidence interval will range from the value 9 to the value

9, meaning there's sort of very, very tight interval.

With the shuffle test, you'll shuffle the labels here, and

do a bunch of experiments, and quickly determine that the p-value's about 0.5,

which is of course not significant, right?

Half the time the difference is 9, the other half it's negative 9,

and so the p-value will be quite high.

So, you always wanna start with the null hypothesis, that there is no difference,

test the significance of that with the shuffle test, and

then measure the confidence interval as a second step.

Okay, so caveats about the bootstrap.

When is it dangerous to use this?

Well, really it is pretty robust.

It actually makes fewer assumptions than a lot of the classical methods make,

as we've described.

It may underestimate the confidence interval for very small samples.

Usually in these data science regimes, in these big data regimes,

very small samples is not the concern that we typically have to worry about.

We often have mini datasets.

Bootstrapping can't be used to estimate the minimum or maximum of a population.

So you should consider why that might be.

So when we're taking averages, it works pretty well.

But the minimum-maximum is very,

very sensitive to outliers, to very local, local values.

If even one more value,

as you scan through the data is higher, then that will be the maximum.

It could have an arbitrarily large effect on the statistic.

And the bootstrap is not going to,

if you take a bootstrap sample with replacement,

you might miss that one outlier that really dramatically changes the value.

And so you'll get a distribution that doesn't match the distribution of

the maximum, okay?

And so more generally, outliers can cause trouble with bootstrapping, but

they sorta can with any method.

So that's not a significant weakness of resampling methods in general,

and in fact, in a moment, we'll see one thing you can do about outliers.

Okay, so it is a little bit sensitive.

Resampling can be done kind of incorrectly with more complex

examples that fail to preserve the original sampling structure.

So we saw this a little bit with the example of the confidence interval just

now, where you need to do a bootstrap sample of one cohort and

a bootstrap sample of the other cohort,

as opposed to a bootstrap sample where you pool them together.

There are ways to actually do it in kind of a pooled way, but in general, you need

to respect that the structure needs to match the experiment you're trying to do.

Okay, another case is if the data are not independent,

you can't use the bootstrap sample.

And it may be tempting to do so, because it may not be obvious that it's not valid.

There are sometimes tricks you can play.

So the trick is if you have a sequence of mutually dependent values,

so that the independence assumption doesn't hold,

what you can do is break those into sub-samples that don't overlap.

So let's say time series of like stock prices or

something, which are not independent, of course, right?

Every value of a stock price depends on the value previously.

Take sub-sequences of these stock prices, non-overlapping.

Break a long history into chunks, and

then treat each chunk as an individual observation.

Compute some statistics on that, and then do resampling on those chunks.

And that's been shown to work, so there are sometimes ways to apply this general

technique in kind of creative ways.

So it's not particularly limited to a few examples,

certainly not the few examples that I'm gonna show.

But think of it as a very general approach.

You know, write a program,

do the simulation to simulate the experiments that you want.

You may have to think creatively about how to apply this in a safe manner, but

it's a very general approach.

As we mentioned, the maximum doesn't work.

And then of course,non-representative samples will always cause trouble, and

they do with the bootstrap as well.

It's not gonna magically cure issues you have with the data.

And you know, one quote here that I copied down that I sort of liked is,

it's tempting to somebody to say, oh, you let the data speak for itself.

And Nate Silver has this quote,

the numbers have no way of speaking for themselves.

There's always gonna be an interpretation about them.

There's always gonna be some knowledge about where that data came from.

There's always gonna be biases lurking in there that only the source of the data

can tell you about, only the context in which it was done can tell you about.

So putting this statistician's hat on as opposed to, say, the machine learning or

computer scientist's hat on, understanding the source of the data,

understanding the experimental design that led to that data.

Or if you weren't aware of the experimental design,

understanding the other kinds of biases or

contexts in which it was sampled is always gonna be crucially important, whether or

not you're using resampling methods or classical methods.

О Coursera

На онлайн-курсах, специализациях и дипломных программах у вас будут первоклассные преподаватели из лучших университетов и учебных заведений мира.

Join a community of 40 million learners from around the world

Earn a skill-based course certificate to apply your knowledge

Gain confidence in your skills and further your career

Coursera делает лучшее в мире образование доступным каждому, предлагая онлайн-курсы от ведущих университетов и организаций.

© Coursera Inc., 2019 Все права защищены.

Загрузить из App Store

Загрузить в Google Play