What is evaluation? Two concepts,
correlation and causality, and an example,
called the important, RAND Health Insurance Experiment.
We'll also cover quantitative versus qualitative evaluation.
So what is evaluation?
Program or policy evaluation is the application of
systematic methods to address questions about policies or programs and their results.
It includes long term monitoring of a program or policy as it evolves,
as well as one shot studies of a single program impact at a given moment.
The approaches used are based on
social science research methodologies and professional standards.
In particular, the field provides processes and tools to obtain valid,
reliable, and credible data to address
a variety of questions about performance of programs.
The whole idea is to provide scientifically measured,
rigorous evidence that informs evidence based policy making or program development.
The central challenge in evaluation however,
is to identify the causal relationship
between the programmer policy and the outcomes of interest.
So, causality is the key.
But what is causality?
Have you heard of the phrase,
"correlation is not causation"?
Let's define correlation first.
Correlation is a relation existing between variables which
occur together in a way not expected on the basis of chance alone.
For example, the obvious correlation
between the month of January and snow in the Northeastern United States.
Causation is a demonstration that one variable directly influences another variable.
When a variable does have an effect on the other variable,
then you can say that you have identified causation.
So, causation is a subset of correlation.
Meaning, you must have correlation to have causation.
But correlation is not equivalent to causation.
Conversely, if you have causation,
you must have correlation.
So, let's give some examples to bring this to life.
Let's start with correlation.
Did you know there is a correlation between the number of
Nobel prizes won by a country and per capita chocolate consumption?
By the way, Switzerland is number one.
Doubtful that the chocolate is causing the Swiss to win all those Nobel Prizes.
Or the correlation between children eating breakfast and getting better grades.
Surprisingly, there is a lot of research that's been done on this.
It may be causal for hungry kids in famine because they can't concentrate.
But it hasn't held writ large in the United States.
A third example, a community's crime rate and its number of police.
Doubtful that the crime rate itself is causing the number of police to go up.
On the other hand, what are the examples of causation?
Smoking and lung cancer.
This one is well established.
Statins use and lowering heart attacks in patients with high cardiovascular risk.
Again, one with a lot of medical evidence.
But what about in health policy?
Let's spend some time going through a canonical example of
a health policy experiment that you should remember.
The RAND Health Insurance Experiment was
a large scale randomized experiment conducted between 1971 and 1982.
For the study, RAND recruited 2,750 families accounting for almost 8,000 individuals.
These individuals were randomized to insurance plans with different designs.
Basically each plan made the individual pay a different share of the costs.
So what did they find? Participants who paid for a share of their health care use
fewer health services than the comparison group given free care.
Cost sharing reduced the use of
both highly effective and less effective services and roughly equal proportions.
Importantly, cost sharing did not
significantly affect the quality of care received by participants.
Wow. Seems like people might actually be okay at figuring out what healthcare is worth.
Well, while cost sharing in general had no adverse effects on participant health,
there were some exceptions.
Free care led to improvements in blood pressure,
dental health and vision.
These improvements were concentrated among the sickest and the poorest patients.
So, then again, maybe cost does matter.
The reason I share this with you is twofold.
First, it illustrates the causal effect of
cost sharing on a number of outcomes like health and use of services,
both high value and low value.
Second, because it is an important health policy finding, that you should know,
it has been the basis for
most health insurance design nationwide for the past 30 plus years.
The main reason for this is that evidence is considered very sound and robust.
That's because randomized trials are the strongest way to build evidence of causation.
But even they have limitations,
more on those tradeoffs later.
Another important dimension of evaluation to highlight is that
not all evaluations are about hard quantitative outcomes like cost or income.
Much of our insight comes from the qualitative.
Qualitative data our key complement to
quantitative impact evaluations because they can
provide added perspectives on a program's performance.
Evaluations that integrate qualitative and quantitative analyses
are characterized as using mixed methods.
So, what are examples of qualitative approaches?
They include surveys, focus groups, interviews, case studies.
Importantly, qualitative evaluations can also be designed to address causality.
Today, we covered what is evaluation,
the concepts of correlation and causality,
examples including the ever important RAND Health Insurance Experiment
and quantitative versus qualitative evaluation.
Christian TerwieschAndrew M. Heller Professor at the Wharton School, Senior Fellow Leonard Davis Institute for Health Economics Co-Director, Mack Institute of Innovation Management
Amol S. Navathe, MD, PhDAssistant Professor of Medical Ethics and Health Policy
David A. Asch, MD, MBAProfessor of Medicine and Professor of Medical Ethics and Health Policy
Roy Rosin, MBAChief Innovation Officer
Kevin Volpp, MD, PhDProfessor of Medicine, Division of Health Policy / Professor of Health Care Management
