узнаете>
Наука о данных>
Анализ данных>
Practical Predictive Analytics: Models and Methods.>
Multiple Hypothesis Testing: Bonferroni and Sidak Corrections

Loading...

Multiple Hypothesis Testing: Bonferroni and Sidak Corrections

Learn the basics of statistical inference, comparing classical methods with resampling methods that allow you to use a simple program to make a rigorous statistical argument. Motivate your study with current topics at the foundations of science: publication bias and reproducibility.

Practical Predictive Analytics: Models and Methods

Вашингтонский университет

Курс 2 из 4 — Наука о больших данных Специализация

Statistical experiment design and analytics are at the heart of data science. In this course you will design statistical experiments and analyze the results using modern methods. You will also explore the common pitfalls in interpreting statistical arguments, especially those associated with big data. Collectively, this course will help you internalize a core set of practical and effective machine learning methods and concepts, and apply them to solve some real world problems. Learning Goals: After completing this course, you will be able to: 1. Design effective experiments and analyze the results 2. Use resampling methods to make clear and bulletproof statistical arguments without invoking esoteric notation 3. Explain and apply a core set of classification methods of increasing complexity (rules, trees, random forests), and associated optimization methods (gradient descent and variants) 4. Explain and apply a set of unsupervised learning concepts and methods 5. Describe the common idioms of large-scale graph analytics, including structural query, traversals and recursive queries, PageRank, and community detection

Просмотреть программу курса

Получаемые навыки

Random Forest, Predictive Analytics, Machine Learning, R Programming

Преподаватели

Bill Howe
Director of Research

Текст видео

[MUSIC]

Okay, so a third potential explanation of this decline effect is

exceedingly simple, but exceedingly common.

And this may be the most important one to learn to apply in your own work when

you're doing statistical analysis.

Okay, and so this is the idea of Multiple Hypothesis Testing.

So the problem here is that if you perform experiments over and over and over again,

you're bound to find something, right?

That's sort of the definition of that.

If you keep rolling dice long enough, you can't just keep rolling dice and

then yell Yahtzee, right?

You get one shot at it.

And the same is true with these experimental design.

Okay, and so this is related to the publication bias problem,

in that you're only showing your positive results, but it's a little bit different.

Because here, you're talking about the same sample, and

you're testing different hypotheses, over the same data.

And so in these situations, either you shouldn't do it at all or if you do

have to it for various reasons, you need to adjust the significance level down.

That means you do not settle for 0.05 as the threshold.

You need to do something much much lower.

Okay.

So, to understand why, consider something pretty basic.

Over completely random data.

And we set the threshold at 0.05, it's alpha at 0.05.

So the probability of detecting an effect where there is none is 0.05.

And the probability of detecting an effect when it exists is 1 minus alpha.

Then a probability of detecting an effect when it exists on every experiment you do

at experiments is 1 minus alpha times 1 minus alpha times 1 miles alpha times 1

minus alpha, assuming that they're independent.

It's okay to multiply probabilities together

if those probabilities are independent.

Finally the probability of detecting an effect where there is none

on at least one experiment is one minus that total, right?

So first build up the probability of being perfect.

And then 1 minus that is the probability of not being perfect without making at

least 1 mistake.

So, if you plot these numbers what you get is, on the X axis here is the number of

tests and the Y axis is the probability of at least one spurious finding.

Right? And making at least one mistake.

Well, it goes up like this.

As you get up to 50 hypothesis tests,

you're up at the 90% chance of at least one spurious finding.

Okay, and so controlling this is known as controlling the Familywise Error Rate.

This is the Familywise Error Rate of at least one mistake.

So this is a pretty stringent constraint.

So how do we correct for that, how do we control the Familywise Error Rate?

Well, one solution is the Bonferroni Correction,

which is just to divide by the number of hypotheses.

Okay, so if your significance level is alpha, 0.05, and you do 20 experiments,

you're testing 20 hypotheses, you just divide 0.05 by 20.

So another correction is the Sidak Correction,

which has this extra condition where the tests need to be independent.

So even though we talked about in the last slide that in order to make that plot,

we were assuming that they were independent, but

the Bonferonni Correction in general does not need to assume that.

Okay, so if you're doing hypotheses test that are related to each other,

you can still do the Bonferroni Correction.

However, to the derive this Sidak Correction,

we're gonna rely on the fact that we're gonna multiply the probabilities together.

Whenever you see probabilities being multiplied together,

that means that you're assuming they're independent.

Okay, so let's see if we can build this up.

[MUSIC]

Рекомендуемые видео

О Coursera

Courses, Specializations, and Online Degrees taught by top instructors from the world's best universities and educational institutions.

Join a community of 40 million learners from around the world

Earn a skill-based course certificate to apply your knowledge

Gain confidence in your skills and further your career

Coursera делает лучшее в мире образование доступным каждому, предлагая онлайн-курсы от ведущих университетов и организаций.

© Coursera Inc., 2019 Все права защищены.

Загрузить из App Store

Загрузить в Google Play