This course provides those involved in educating members of the health professions an asynchronous, interdisciplinary, and interactive way to obtain, expand, and improve their teaching skills. These skills can then be applied within their own professional context, with a variety of learners, extending across many stages. After completing this course, learners will: 1. Understand educational theory as it relates to health professions education 2. Match instructional methods with desired educational outcomes 3. Learn a variety of applied teaching techniques 4. Share successful teaching strategies Suggested resources will include educational videos, individual readings, recommended reference books, and crowd-sourced recommendations. All students should have dependable access to the internet with video streaming capabilities. Some students may want the ability to utilize on-line video conferencing as well.
03.01 - Multiple-Choice Question Writing
Loading...
From the course by University of Michigan
Instructional Methods in Health Professions Education
107 оценки
From the lesson
Unit 3
In this unit you will learn about instructional design and individual assessment, including multiple-choice question writing, skill assessment, oral presentation, and rubrics and standardization.
Meet the Instructors
Caren Stalburg, MD, MAAssistant Professor, Obstetrics & Gynecology
Medical School
Hi,
welcome back.
Today we're going to talk about multiple choice question writing.
But before we talk about assessments, I want to just review for a
moment the idea that we need to put the whole picture together.
So our intended learning outcomes; what we're trying to
teach and how we're actually going to assess it.
So just as a reminder I want to talk, just
for a moment about how we should blueprint our curriculum.
And then switch to an assessment.
Specifically multiple-choice questions.
The rest of this module, we'll talk about other
forms of assessments and those will be in separate videos.
And then the challenge will be to
construct multiple-choice questions to assess higher-order learning.
So those are the goals of this session for you.
So people always say, so you want to teach.
and it, it can be a very
[LAUGH]
daunting task case in point, right here.
but really, when you're trying to teach something,
there's lots of questions that you need to ask.
So you need to ask, what are you going to teach?
And where might you teach it?
Whom is the learner, how are they going to learn, but most
importantly, after you've done all of the instruction and everything else.
How will you know that they actually learned
what it is you set them out to learn.
So one way to try to know
where you're heading is to create something called a curriculum blueprint.
Just like you might do a blueprint for a building you're trying to build.
So I always say to start backwards.
So you define your intended learning outcomes before you start.
And that's why you'll see in these videos I always start with
ILOs, as well as online, the sessions always start with the ILOs.
So we can always keep track of where
we're headed. Okay.
Once you know where you're headed, then you can
figure out why you're doing it, and what to do.
So the first question is, well, really who cares what you're trying to teach.
And so, you may want to consider doing a needs assessment.
Is this something that people want to learn or need to learn?
And then the important thing is to also define your learner.
Are we talking about novices? Are we talking about experts?
Are we talking about a global audience?
that helps define which way you want your curriculum to go.
And then again the most important piece for this session and that what
we're doing in this whole course is what teaching tools will you use.
So my favorite, sort of illiteration, is to pick a
pedagogy and you want that pedagogy to match your intended outcomes.
But first you have to think about well, okay, I'm going to teach something.
And I'm going to try and measure their success, but how do I do that?
So while, we will I promise get to pedagogies and specifically what
you can use in which situation, we need to talk just a minute about assessments.
Which is why this module is going to be about multiple choice questions.
So the first obvious question or statement is
that there's no such thing as a perfect assessment.
You're trying to get an approximation.
And so you'll hear terms like reliability, meaning that the results
you get when you administer an examination or a test of some kind, are reproduceable.
They can be relied upon, that People who
have learned this information to this degree will reproduce
the same results in a variety of tests, situations.
So, if you gave a similar test to the one you just gave.
If you were having a reliable assessment, you would expect that the results
on a similar test would be the same to the one that you're giving.
Now the challenge with reliability is that
it's going to be limited by sampling errors.
So if you don't have enough items on your test, then you may
actually not be able to capture all of the information that your learner has learned.
Or if you have different examiners especially in a
standardized test situation, or an essay, or an oral presentation.
If there's variation between the examiners, there may not be reliability
between examiner A and examiner B. So examiner B may grade tougher, or
more rigorously, than examiner A. And so one way to sort of
mitigate that is to have your examiners actually grade the same tests.
So that you can make sure that there's
standardization between how they look at an exam.
So for instance, you may have a 100 papers that you
need to grade and you have three people who are grading.
So those three people, you could have them actually grade the same 10 items.
Sorry, the same 10 exams, and then see how closely they each graded.
That exam to the same level and once you know that they can
grade the same way then you have something called, sort of, inter rater reliability.
So that you know no matter who gets the exam
paper, that person has been trained and standardized
to score the exam in the same way.
Another challenge is something called validity which is a little
hard to sort of get your head around a little bit.
But really what validity means is that
you're actually measuring what you think you're measuring.
So basically you can have two, there's lots of different types of validity.
and you can have an entire course on just these two
issues of reliability and validity but I just want to give an overview.
So you can have content validity, which basically says, the content on the
exam is what I either taught or what I think is crucial for people to learn.
And if you don't sample enough of the
curriculum, then you may not have enough content validity.
So,
an example that many people us is, that if you're
giving an exam on pulmonology or diseases of the lungs.
You wouldn't want all of your questions to be about pneumonia.
You may want to have other questions about
pneumonia and asthma and reactivate airway disease, etc.
The other piece that's hard to understand is something called construct validity.
So with construct validity You're basically
measuring something that is a theoretical thing.
So, for instance, how can you measure intelligence, right?
If, you many have a theoretical construct of what intelligence is.
And so then you create an exam to try to measure that.
So if your exam actually doesn't match up with the construct,
then you're not measuring what you think you're measuring.
So you have to just be really careful in terms of
assessments to make sure that the results that you get are reproducible.
Since you can rely on the exam to test what it is you think you're testing.
And also that your results are going to be valid.
'Kay, that's a lot.
But another way to think about it, another way to organize the
structures, is to think, okay, what are my intended learning outcomes.
And you can go back to doing taxonomy and look at those
learning verbs depending on the level that you want people to attain.
So if you want them to be able
to synthesize, you might use synthesize verbs Etcetera.
Or to work at Miller's pyramid.
And think about the domain that you're trying to teach.
Is it a knowledge, is it a skill, or an attitude?
Again we want to match that
pedagogy to our intended learning outcome as well as
to the type of domain, and the assessment should match.
So let's flip it just a little bit differently.
Let's say, all right, we're talking about assessment
types and we're talking about the variety of domains.
So we can talk about knowledge domains and we
can talk about how you would evaluate someone's knowledge.
You could use multiple-choice questions.
You could use short answers or problem-solving sets.
You could have them do essays. You could have them do oral exams.
You can see how each one of those assessments will match
or map back differently to which ever level on Bloom's Taxonomy.
Or Miller's pyramid you're intending to get
to at the end of your educational intervention.
the other way to think about it is okay, how do I evaluate skills?
And many of us are familiar
with OSCEs, or Objective Standardized Clinical Examinations.
Or OSATS, which are Objective Standardized Assessments of Topical Skills.
Or in the United Kingdom, they're also called
DOPS, which is Directed Observation of Procedural Skills.
Now the hardest domain to measure are attitudes or attributes that people have.
And you can use essays, portfolios, art,
either written or performed, and literature as well.
So again, I'm just giving a basic overview
of why thinking about ILOs And thinking about
how you're going to assess what it is you're
trying to get your learner to, will then help
you pick your pedagogy.
So it's a sort of circular, you know, which came first, the chicken or the egg.
but it's something that I wanted to make sure we mentioned.
So let's get really to the task at
hand, which is talking about multiple choice questions.
So, the biggest question is, what's a multiple choice question?
Is it A, a way to assess factual knowledge?
B, an efficient
way to assess individuals? C, a way to assess higher order
understanding of information? D, a question with only one right answer.
Or E, all of the above. Okay, let's go through first
the types of multiple choice questions.
So you can have a multiple choice question
that is basically a true, false item format.
The important piece about a true, false
item Is that the choices must be unambiguous.
And so you can't have things that could
somehow be true or might possibly be false.
There have to be clearly false answers and
clearly right answers.
So, for instance, you could not say that the sky is purple Or blue-gray or blue,
because depending on where you are and what time of day and the light, et cetera.
and the location, it may be that the sky could be all of those things.
So the trick with true-false items and especially
to keep yourself out of trouble after you've
administered exams.
To make sure that those choices are truly
true and the false choices are absolutely false.
The challenge with these types of multiple-choice questions, though, is
that they lend themselves towards the recall of an isolated fact.
And so if you're trying to asses higher order skills, you
may not sort of pick that up by doing a true/false item.
So the other item format is single best option format and that allows for
the best matched answer to be produced by your learner.
And you can incorporate multiple cognitive jumps within the question.
Meaning, you can ask a straight forward
knowledge question like the capital of Iowa is?
or you can ask other questions that require the
learner to sort of make a few cognitive jumps.
So you
could ask a question the city where London Bridge is located?
Is which of the following, so that, that
makes the person sort of jump to the answer.
Okay.
so let's go through sort of the anatomy of a multiple choice question.
But before we do that I just want to
place out a little disclaimer there which is that.
I have the opportunity to write multiple choice
questions for the National Board of Medical Examiner in
Philadelphia, PA.
And most of what I'm talking about I learned
from them and I write questions for them as well.
So let's look at the anatomy of a multiple-choice question.
So the first part.
An instructor's teaching a module
on designing assessments for distance-learning.
There are more than 5000 students.
She wants to confirm that students
learn the components of a multiple-choice question.
Which of the following assessment types would be most suited to her purpose.
A Essays, B Group projects, C Video-taped
submissions, D Multiple choice question examinations.
Okay before you panic, we're actually just
[LAUGH]
going to go through the different components of this question.
So the beginning, or the first paragraph is what you call the item stem;
sort of sets the, the scenario that you want people to think about.
The next piece is the question that you're asking.
So it's the lead-in question.
And then the remaining pieces, of course,
are the answer, the one correct answer, and
distractors.
Okay, so the item stem should really be focused
on a concept and should not be based on memorization.
Many of us are familiar with the item stem being in the form of a clinical vignette.
And what you want your items stem to do is have enough information that
you can formulate an answer to the
question without ever looking at the options.
So here's an example.
A 35-year-old
woman complaints of nausea and vomiting for the past three weeks.
She also notes urinary frequency without burning.
Her last menses was seven weeks ago. On examination, she is afebrile.
Notable finding on exam includes a mildy enlarged pelvic mass.
Her Bhcg is 6, 582.
Now hopefully most of you with multiple
cues within that item stem, know the answer
before I even give you the options.
So the next piece of the question,
the multiple choice question's actually the lead-in.
And the lead in should be specific to the vignette.
So after that vignette which I've included on the slide, the
next question is which of the following is the most likely diagnosis?
The important thing about the lead-in question is that
it can't be answered simply by reading the lead-in alone.
Because otherwise you have no need for the vignette.
So the lead-in should really direct the examinee towards a thought
process or what, of what it is you're trying to ask them.
So what is the
[INAUDIBLE]
of the question basically. Okay, now the answers and distractors.
Again I've included the same stem, and I've included the same lead in questions.
And now I've given you some answers and distractors.
A, Leiomyoma, B pregnancy, C small bowel
obstruction, D Ovarian cancer, or E Colon cancer.
Now, important things to remember is
that answers and the distractors should be similar types.
So there's all diagnoses that I've listed
and all diagnoses of organs within the pelvis.
I did try to alphabetize
[LAUGH]
them, but I had failed in that effort.
one of standardization is just to alphabetize them, since you can
avoid people sort of being test wise and always picking answer C.
The other thing you want to do is avoid something called queuing where if you
include only two gynecology answers. And everything else is
different that may cue. Or if you include similar words in the
distractors that may cue people towards choosing between those two.
And then they just have a 50% chance of getting
The question right, wh, rather than actually answering the question.
sometimes your correct answer will be about a mile long.
And that actually cues people that that's the right answer.
The other thing to remember is that all of the choices
that you put as distractors should be plausible to the vignette.
So in this case, I wouldn't put lung cancer,
because that's just implausible given the vignette as it's written.
'Kay, so here's my own example of poorly-written stem.
The instructor is teaching a module on designing assessments
for distance learning There are more than 5,000 students.
She wants to confirm that students learn from
the session. Which of the following assessment types?
Now, the problem with the stem is that,
what are students supposed to learn from the session?
Right, if you don't know what the intended outcome is.
If we don't know what they're supposed to learn, how
can we pick from the variety of assessments at all.
So that's actually an example of a poorly written stem.
This is an example of a poorly written lead-in.
So it's sort of the
same stem, changed a little bit, as that she wants to
confirm that her students learn the
components of a multiple-choice question And
then the lead-in says, which of the following assessment types is
best for testing memory of the specific components of a multiple-choice question?
Again, you don't need the vignette.
You can put your hands over the vignette and just read the lead in question.
So that's poorly written. To continue on
this journey of poorly written questions.
These distractors are actually also not the best.
And the reason why is, they're kind of vague.
One of them is very, very long and that may sort of
move people towards the right answer because D is the longest answer.
Even if they don't know that an essay could
be a good way to understand the components of a multiple choice question.
Because in theory I could ask you to write an essay about, why
do you need to have good distractors or what would make a good distractor.
So many of these other options could fit.
So as I mentioned I actually write
questions for the National Board of Medical Examiners.
And they have a really helpful resource
online that's a complete item writing manual
that talks about many of the things we've talked about today.
But if you can write a really good multiple choice question,
you actually can assess learners in a really efficient and cost effective.
and pretty reliable and valid way.
Alright, see you again soon. Thanks.
Ознакомьтесь с нашим каталогом
Присоединяйтесь бесплатно и получайте персонализированные рекомендации, обновления и предложения.
Coursera делает лучшее в мире образование доступным каждому, предлагая онлайн-курсы от ведущих университетов и организаций.
© Coursera Inc., 2018 Все права защищены.
