Multiple Hypothesis Testing: Bonferroni and Sidak Corrections

Loading...

Из курса от партнера University of Washington

Practical Predictive Analytics: Models and Methods

275 оценки

University of Washington

Practical Predictive Analytics: Models and Methods

275 оценки

Курс 2 из 4 — Specialization Data Science at Scale

Statistical experiment design and analytics are at the heart of data science. In this course you will design statistical experiments and analyze the results using modern methods. You will also explore the common pitfalls in interpreting statistical arguments, especially those associated with big data. Collectively, this course will help you internalize a core set of practical and effective machine learning methods and concepts, and apply them to solve some real world problems. Learning Goals: After completing this course, you will be able to: 1. Design effective experiments and analyze the results 2. Use resampling methods to make clear and bulletproof statistical arguments without invoking esoteric notation 3. Explain and apply a core set of classification methods of increasing complexity (rules, trees, random forests), and associated optimization methods (gradient descent and variants) 4. Explain and apply a set of unsupervised learning concepts and methods 5. Describe the common idioms of large-scale graph analytics, including structural query, traversals and recursive queries, PageRank, and community detection

Из урока

Practical Statistical Inference

Learn the basics of statistical inference, comparing classical methods with resampling methods that allow you to use a simple program to make a rigorous statistical argument. Motivate your study with current topics at the foundations of science: publication bias and reproducibility.

Multiple Hypothesis Testing: Bonferroni and Sidak Corrections3:55

Multiple Hypothesis Testing: False Discovery Rate4:07

Multiple Hypothesis Testing: Benjamini-Hochberg Procedure3:05

Big Data and Spurious Correlations4:52

Spurious Correlations: Stock Price Example3:15

How is Big Data Different?3:40

Познакомьтесь с преподавателями

Bill Howe
Director of Research
Scalable Data Analytics

[MUSIC]

Okay, so a third potential explanation of this decline effect is

exceedingly simple, but exceedingly common.

And this may be the most important one to learn to apply in your own work when

you're doing statistical analysis.

Okay, and so this is the idea of Multiple Hypothesis Testing.

So the problem here is that if you perform experiments over and over and over again,

you're bound to find something, right?

That's sort of the definition of that.

If you keep rolling dice long enough, you can't just keep rolling dice and

then yell Yahtzee, right?

You get one shot at it.

And the same is true with these experimental design.

Okay, and so this is related to the publication bias problem,

in that you're only showing your positive results, but it's a little bit different.

Because here, you're talking about the same sample, and

you're testing different hypotheses, over the same data.

And so in these situations, either you shouldn't do it at all or if you do

have to it for various reasons, you need to adjust the significance level down.

That means you do not settle for 0.05 as the threshold.

You need to do something much much lower.

Okay.

So, to understand why, consider something pretty basic.

Over completely random data.

And we set the threshold at 0.05, it's alpha at 0.05.

So the probability of detecting an effect where there is none is 0.05.

And the probability of detecting an effect when it exists is 1 minus alpha.

Then a probability of detecting an effect when it exists on every experiment you do

at experiments is 1 minus alpha times 1 minus alpha times 1 miles alpha times 1

minus alpha, assuming that they're independent.

It's okay to multiply probabilities together

if those probabilities are independent.

Finally the probability of detecting an effect where there is none

on at least one experiment is one minus that total, right?

So first build up the probability of being perfect.

And then 1 minus that is the probability of not being perfect without making at

least 1 mistake.

So, if you plot these numbers what you get is, on the X axis here is the number of

tests and the Y axis is the probability of at least one spurious finding.

Right? And making at least one mistake.

Well, it goes up like this.

As you get up to 50 hypothesis tests,

you're up at the 90% chance of at least one spurious finding.

Okay, and so controlling this is known as controlling the Familywise Error Rate.

This is the Familywise Error Rate of at least one mistake.

So this is a pretty stringent constraint.

So how do we correct for that, how do we control the Familywise Error Rate?

Well, one solution is the Bonferroni Correction,

which is just to divide by the number of hypotheses.

Okay, so if your significance level is alpha, 0.05, and you do 20 experiments,

you're testing 20 hypotheses, you just divide 0.05 by 20.

So another correction is the Sidak Correction,

which has this extra condition where the tests need to be independent.

So even though we talked about in the last slide that in order to make that plot,

we were assuming that they were independent, but

the Bonferonni Correction in general does not need to assume that.

Okay, so if you're doing hypotheses test that are related to each other,

you can still do the Bonferroni Correction.

However, to the derive this Sidak Correction,

we're gonna rely on the fact that we're gonna multiply the probabilities together.

Whenever you see probabilities being multiplied together,

that means that you're assuming they're independent.

Okay, so let's see if we can build this up.

[MUSIC]

Ознакомьтесь с нашим каталогом

Присоединяйтесь бесплатно и получайте персонализированные рекомендации, обновления и предложения.

Coursera делает лучшее в мире образование доступным каждому, предлагая онлайн-курсы от ведущих университетов и организаций.

© Coursera Inc., 2018 Все права защищены.

Загрузить из App Store

Загрузить в Google Play