Bad Science Revisited: Publication Bias

Loading...

Из курса от партнера University of Washington

Practical Predictive Analytics: Models and Methods

275 оценки

University of Washington

Practical Predictive Analytics: Models and Methods

275 оценки

Курс 2 из 4 — Specialization Data Science at Scale

Statistical experiment design and analytics are at the heart of data science. In this course you will design statistical experiments and analyze the results using modern methods. You will also explore the common pitfalls in interpreting statistical arguments, especially those associated with big data. Collectively, this course will help you internalize a core set of practical and effective machine learning methods and concepts, and apply them to solve some real world problems. Learning Goals: After completing this course, you will be able to: 1. Design effective experiments and analyze the results 2. Use resampling methods to make clear and bulletproof statistical arguments without invoking esoteric notation 3. Explain and apply a core set of classification methods of increasing complexity (rules, trees, random forests), and associated optimization methods (gradient descent and variants) 4. Explain and apply a set of unsupervised learning concepts and methods 5. Describe the common idioms of large-scale graph analytics, including structural query, traversals and recursive queries, PageRank, and community detection

Из урока

Practical Statistical Inference

Learn the basics of statistical inference, comparing classical methods with resampling methods that allow you to use a simple program to make a rigorous statistical argument. Motivate your study with current topics at the foundations of science: publication bias and reproducibility.

Outliers and Rank Transformation3:24

Example: Chi-Squared Test3:39

Bad Science Revisited: Publication Bias4:47

Effect Size4:26

Meta-analysis5:05

Fraud and Benford's Law4:09

Intuition for Benford's Law2:50

Benford's Law Explained Visually3:50

Познакомьтесь с преподавателями

Bill Howe
Director of Research
Scalable Data Analytics

[MUSIC]

So now that you're armed with a little bit of basic terminology,

let's go back to this first slide from this New York article.

And so the question we raised was, what accounts for

this truth wearing off effect?

How can we explain what's going on?

Okay?

So one reason is publication bias.

All right, so let me read you a couple of quotes from this article about

publication bias.

So, in the last few years, several meta-analyses,

and we'll talk about what a meta-analysis is in a little bit,

have reappraised the efficacy and safety of antidepressants and concluded that

the therapeutic value of these drugs may have been significantly overestimated.

Although public, and there's other examples in this article as well.

So go back to review the literature and find out that things have been overstated.

Why?

So, although publication bias has been documented in the literature for decades,

and origins and consequences are debated extensively.

There is evidence suggesting that this bias is increasing, all right, so

I haven't told you what publication bias is yet, but

you may be familiar with the concept and see some of the effects.

So a case in point in the field of biomedical research in autism

spectrum disorder,

which suggests that in some areas negative results are completely absent.

All right. So, what does that mean?

That means that you're only publishing papers that show significant positive

gains.

If we try several treatments and none of them work except for

one, we try 20 treatments and only 1 works, how many papers do we publish?

One, not 20.

Okay.

So, how is this a problem?

So, do we have an explanation for this decline effect with publication bias?

So, how does this actually work?

Well let's make a plot where this study size is on the x axis and

notice this is log scale.

This is 10 and this is 100 and this is 1000 and so on.

Right?

Well study size meaning the number of patients that are involved in the study.

So the bigger the study size, the more statistical power you have.

And we'll define what statistical power means precisely in a bit.

But the better you are able to determine actual effects, right.

And the assumption here is perhaps that, you know, as time goes on and

you see some results, you are able to, you or other researchers,

are able to garner more money, more funding to do larger and larger studies.

So this is maybe phase one, phase two, phase three trials of some new drug.

They get bigger and bigger and bigger sets of patients as you get more and

more momentum behind it.

And this data is not real.

This data's simulated.

But imagine you see this kind of decline effect where the results, okay,

well sorry the y axis.

The y axis is the effect size, and

we'll talk about what the effect size is precisely in a little while.

But, this is the degree of positive outcome, let's say.

All right, and let's say negative is bad, and positive is good.

So, this is the number of smokers you were able to convince to quit,

with some intervention counseling method.

Or, the number of white blood cells

increased as a result of some treatment or so on.

Okay, so positive is good.

Well, this decline effect, let's imagine, shows this sort of a pattern, right,

where the early studies with just ten participants is up here,

and as the study went up it sort of got worse and worse.

This is the effect that we see, how do we explain this?

Well, this kind of an effect would

be directly explainable just by publication bias.

Imagine that every dot now is a test that was done by some group somewhere for

this phenomenon what you'd expect is this kind of funnel shape where

the studies get more and more accurate, as you get larger, and larger, and larger.

All right, and they will, this is, you can't get around this, right.

As the study size goes up, you do have more statistical power.

You're able to better discriminate real effects, from false effects, and so on.

But you'll notice, that the actual effect that it's regressing to here is 0.0,

there is no effect.

And yet, of course you're going to get some, just due to variability out here.

And so if you only report the positive ones,

you'll end up with this mysterious decline effect.

[MUSIC]

Ознакомьтесь с нашим каталогом

Присоединяйтесь бесплатно и получайте персонализированные рекомендации, обновления и предложения.

Coursera делает лучшее в мире образование доступным каждому, предлагая онлайн-курсы от ведущих университетов и организаций.

© Coursera Inc., 2018 Все права защищены.

Загрузить из App Store

Загрузить в Google Play