Hi, in this module I'm going introduce type II errors to you,
or for some of you this may be a review.
Type II error refers to the phenomenon of incorrectly concluding that there
is no relationship or no effect when in fact there is one.
In other words, we fail to reject the null hypothesis even when we should reject it.
So it's the opposite of the Type I errors that we just talked about.
A type I error, again,
is a situation where we have a false positive.
We conclude that there is a relationship,
a effect, even more,
even when there actually isn't one.
So here, there is an effect,
or there is a relationship,
but we don't detect it.
So we failed to deduct a difference or
relationship that actually exists out there in the population.
If only we could measure it properly.
Statistical power refers to the ability to detect a difference or a relationship.
You'll hear the term 'statistical power' quite often,
especially if you take a statistics class taught in
Psychology or Epidemiology or certain other fields where they
are more concerned or more adept at dealing with type II errors.
And you'll often see sometimes people write a Beta,
the Greek letter Beta,
as shorthand for the probability of a type II error.
So, one minus beta is our statistical power.
So just to give you a few examples of what a type II error might be.
Maybe a situation where we're testing the effect of some drug,
and actually maybe the drug does have an effect.
But when we do our test,
we don't find that effect,
or there is a relationship between, say,
education and income that we don't pick up or at
least we don't regard as statistically significant.
So I mentioned that one minus Beta is the measure of statistical power.
So quite often, again,
if you take statistics from psychologists or in certain other fields,
we'll talk a lot about this because they get to design
their studies themselves and they
decide on how many subjects they're going to include in their studies.
So they figure out in advance what kind of statistical power they
need or what level of significance they want to be able to discern and affect that.
And then they carry out what are called 'power
calculations' to work out the size of the study,
the number of subjects they'll need in order to properly
resolve whether or not an effect exists and therefore,
hopefully, avoid a type II error.
Now, how do people,
when they're worrying about this,
reduce the chances of a type II error?
Well, they can increase the sample size,
that increases statistical power.
So we mentioned that a larger sample size
generally yields larger test statistics which reduce our P value.
So, again, less likely do commit a type II error with a larger sample,
less likely do miss an effect that actually is there.
If we are working with secondary data where we don't have any control over
the size of the sample because somebody else collected the data,
and we inherited it or accessed it,
we can try pooling data.
So increasingly some people try to pool data from, perhaps,
multiple waves of the same study or from multiple studies that cover the same topic,
and that gives them more cases, more statistical power.
Power that helps them detect possibly relationships that they might otherwise miss.
They can always loosen the criterion for the hypothesis tests,
so you settle for a criterion of point five for the significance instead of point 01.
Now, that may be a bit risky because of course it increases your chances of a type I error
when you loosen the criterion for the test.
And again, something that's always
a good idea you can try to reduce the measurement error.
So a measurement error,
variability in the measurement of the outcome,
and the other variables, can,
by introducing noise into our measurements,
change or alter our test statistics and
make it more likely that we'll end up committing a type II error that is,
or missing effect that actually is there.
So, what are some of the implications of
the prospect or the possibility of type II errors?
One important one, and I see people
violating this principle actually on a fairly regular basis,
is that statistical significance, it helps prove,
that or at least helps provide evidence that a relationship or a difference exists.
But the reverse is not true.
Lack of statistical significance does not rule out the existence of a relationship.
So, lack of statistical significance should be interpreted as
a relationship unproven or relationship not proven in Scottish jurisprudence,
but it's not a relationship disproven.
So sometimes I see people making presentations and they'll report,
say regression coefficients, that actually have large magnitude,
or effect sizes that appear very large but actually
the P values make it clear that the effect is not statistically significant.
And then they go on to say, 'well, in fact,
the result is not statistically significant,
therefore, there is no relationship'.
So again, a large P value is not a proof that a relationship doesn't exist,
it just means that the data at hand is not sufficient to prove that it exists.
There may be a relationship but our sample may be too
small or there may be too much noise or our criterion,
in some cases, may not be too strict.
I wouldn't advise loosening criterion but never hurts to try to gather more data.
So, that would be the one thing that I would ask you to keep in mind.
On the one hand, if you are going into psychology or
certain other fields where you design studies yourselves, experimental studies,
you'll have to learn about statistical power,
about power calculations to work out what kind of sample size you're going to
need to measure the effects that you hypothesize are there.
But for the rest of us,
I think the practical implication of thinking about
Type II errors in the meaning of P values,
especially values that don't appear statistically significant,
is that if something isn't statistically significant
that doesn't prove the relationship isn't there,
i it just means we can't reject
the null hypothesis and we don't have evidence of a relationship.
You want to disprove a relationship,
if you want to rule out a relationship,
need to show that there is a small magnitude or a small coefficient size,
and you have to have a large sample to convince
people that you're really decisively ruling it out.
Cameron CampbellProfessor of Social Science
