Improving Your Statistical Questions

One of the biggest improvements most researchers can make is to more clearly specify their statistical questions. When you perform a study, what is it you really want to know? What are different types of questions we can ask? Which question does a hypothesis test really answer, and is this answer actually what you are interested in, or is the question you are asking more about exploration, description, or prediction? How can we make riskier predictions than null-hypothesis tests, and why is this useful?

There is little use in making predictions if you can never be wrong - so how do we make sure your predictions are falsifiable? We discuss why falsifiable predictions are important, and how to make your predictions falsifiable in practice. One important aspect of making predictions falsifiable is to specify a range of values that is not predicted, and we will examine different approaches to specifying a smallest effect size of interest.

If studies are designed to answer a question, you should make sure the answer you will get after collecting data is informative. Instead of mindlessly setting Type 1 and Type 2 error rates, we will learn why it is important to be able to justify error rates, and some approaches how to do so. We discuss the benefits of using your smallest effect size of interest in power analyses, and why learning to simulate data is a useful tool. Simulations can help you to improve your understanding of statistics, enable you to design informative studies, and even ask novel questions.

Regrettably we work in a scientific enterprise where the published literature does not reflect real research. Publication bias and selection biases lead to a scientific literature that can’t be interpreted without taking these biases into account. We will discuss what real research lines look like, and how to meta-analytically evaluate the literature while keeping bias in mind.