This course presents research conducted to increase our understanding of how data collection decisions affect survey errors. This is not a “how–to-do-it” course on data collection, but instead reviews the literature on survey design decisions and data quality in order to sensitize learners to how alternative survey designs might impact the data obtained from those surveys. The course reviews a range of survey data collection methods that are both interview-based (face-to-face and telephone) and self-administered (paper questionnaires that are mailed and those that are implemented online, i.e. as web surveys). Mixed mode designs are also covered as well as several hybrid modes for collecting sensitive information e.g., self-administering the sensitive questions in what is otherwise a face-to-face interview. The course also covers newer methods such as mobile web and SMS (text message) interviews, and examines alternative data sources such as social media. It concentrates on the impact these techniques have on the quality of survey data, including error from measurement, nonresponse, and coverage, and assesses the tradeoffs between these error sources when researchers choose a mode or survey design.
4.1.2 Mobile Web Surveys
Loading...
From the course by University of Michigan
Data Collection: Online, Telephone and Face-to-face
118 оценки
University of Michigan
118 оценки
Course 2 of 7 in the Specialization Survey Data Collection and Analytics
From the lesson
Module 4: Emerging modes, new data sources
In this lesson, we focus on some new data collection modes such as mobile web surveys and SMS text interviews, as well as alternative data sources such as sensor data, administrative data, and social media. By the end of this lesson, you will have a sense of the issues to which survey methodologists and survey researchers are devoting much of their attention these days. You will be able to weigh the pros and cons of these new methods and data sources.
Meet the Instructors
Frederick Conrad, Ph.D.Research Professor, Survey Methodology
Institute for Social Research
Welcome back, we're continuing with our discussion of Mobile Web surveys and
focusing now on measurement error in Mobile Web.
One issue is that Mobile Web is actually mobile.
So people can complete questionnaires really anywhere,
including away from home, outdoors, while traveling,
hopefully not driving and this is potentially a double edge sword.
So while this may be more convenient, allowing people to
complete the questionnaire in locations that work for them.
It may also inhibit honest responding to sensitive questions, because
people may find themselves in context and social context which are not private.
And so the chances that somebody might see their answers to sensitive questions,
as they are answering them is potentially increased and
this could potentially discourage giving undesirable answer.
Mavletova and
Couper looked into the context in which Web surveys are completed.
And they found that 45% of their respondents filled out the questionnaire
outside their home versus 29% who did this with a conventional computer or laptop.
They still found only small differences in responses to sensitive questions
despite the increase completion away from home.
So, they're finding anyways that mobility doesn't
inhibit responding of candidly insensitive questions.
These are data from Antoun who did an experiment on the Dutch list panel
in which the same respondents completed the questionnaire once on
the mobile device and once on a conventional computer or laptop.
And he found that those using a mobile device were responded away from home
more often, they were walking or moving more often.
They changed locations more often.
They engaged in multitasking more often.
No reliable effect, no significant effect of being distracted by other things.
But the mobile respondents when respondents were mobile,
they found that they were around other people more often.
So, the mobile respondents really are more mobile than the PC respondents.
The question is how did this affect responses to sensitive questions?
As we'll see in a moment, Antoun also like Mavletova and Couper found no
evidence that mobility inhibited Honest responding to the sensitive questions.
One design feature that has been explored quite a bit, we're still in the early
days of evaluating Mobile Web, but even in the early days,
this feature paging versus scrolling has been examined in a number of studies.
And so the idea is that with a paging design, there's usually just one or
maybe two questions per screen, per page with the scrolling design.
The entire questionnaire or at least many questions can be contained on what's in
effect one page or one screen, but to see all the questions.
The respondent needs to scroll generally, vertically.
So with mobile devices, Mavletova and
Couper found that paging design took more time than a scrolling design.
9.1 minutes to complete the questionnaire with paging versus really
about half that time.
4.5 minutes to complete the questionnaire by scrolling.
The DeBruijne and Wejnant found a very similar effect.
59 versus 43 seconds per question for the paging design versus the scrolling design.
So again, scrolling seems to be faster than paging.
Now the explanation in Mavletova and Couper, and also in DeBruijne,
and Wejnant is that this could well be due to download time.
It might not be that it's a usability issue.
It might not be that users, respondents are actually slower to page on to scroll,
but that the paging design requires more downloads.
A download for every new page and given the speed of the connection, that
could be responsible for the increased time for paging versus scrolling.
Whatever the reason for the slower completion or
the longer completion times in paging, it doesn't seem to affect response quality.
The DeBruijne and Wejnant have no effect on the response distributions
from this distinction, and design, and
no evidence that primacy is responsible for faster times with scrolling.
So you might imagine that with a scrolling design,
respondents are just picking the first option and
then moving to the next question, but they found no evidence for that.
Also, Peytchev and Hill,
whose study was done a few years earlier than the two we've just been discussing
found no effect of a different kind of scrolling on the response distributions.
So in their design, they presented a question with five response options,
a scale of five options.
In one design, only three were visible on the screen at one time.
In the other, all five were visible and they asked the question whether seeing
only three options would lead to a preference or biased for
those options that were initially visible in that design over the ones that
required scrolling to be displayed.
They found no evidence of that either.
Some of the respondents were willing to expose the final two options,
the rightmost options often enough, so that there were no differences.
There was no bias toward the options that were in this initially visible compared to
the condition of which all five were visible.
A general theme coming out of these studies is that the response
process is quite similar in Mobile Web and Conventional web, PC Web.
It even seems to be similar to the response process observed in other modes.
This is the finding of Peytchev and Hill who tried to replicate well-known
response effects that is phenomena that were observed,
generally differences between two designs that were observed in other modes.
They attempted to replicate these in Mobile Web,
just to see if the response prosthesis were comparable.
So for example, they replicated the well-known question order of context
affect, published by Hyman and Sheatsley.
In the original study, they were more likely to say the that US should allow at
the time, the 1950s, the Cold War was underway should allow communist
reporters to enter the US and report the news as they see it.
If they were first asked a question about whether a US reporter should
be allowed into Russia to report the news back.
If they weren't asked that question first, they were much less willing to consent to
have a quote, communist reporter enter the US and report the news.
So, the order of the questions mattered is the point.
In Peytchev and Hill's attempted replication of this study,
they updated it and they ask two questions.
One was about the Iranian President Ahmadinejad speaking in the US and
the other was about US President Clinton
speaking in an authoritarian country like Iran.
What they found was that there was about 50% support for
Ahmadinejad speaking in the US, whether was asked first or second.
The question about Clinton, if asked first led to 84% approval.
So, 84% of the respondents approved
of Clinton speaking in an authoritarian country.
However, if they had previously been asked about
Ahmadinejad where half were opposed to his speaking in the US.
Only 57% felt that Clinton should speak in an authoritarian country.
So, they post a norm of reciprocity in the original study and
it's essentially replicated here on mobile devices.
Hill also successfully replicated well-known study by Schwarz and
his colleagues, which asked people to report on
the frequency with which they watch television using two response scales.
What Schwarz and colleagues called a scale with low-frequency alternatives and
high-frequency alternatives.
And as you can see, they are actually very similar, but
the low-frequency alternative scale starts with an increment of half an hour.
The high-frequency alternative scale starts with an increment of up
to 2.5 hours, then both scales proceed in increments of half and hour.
What Schwarz and his colleagues found was that the low frequency scale,
more people picked the response option that was less than two and
a half hours than in the high-frequency scale.
And they attribute this to respondents really not paying attention to
the numerical properties of the scale, but picking a position in the scale.
So for example, if a respondent reasoned well,
I watch about as much TV as most people, so I'll pick the middle option,
that would lead to very different numbers numerical values depending on the scale.
This is exactly what Peytchev and Hill replicated replicated,
what they observed when presenting this on a mobile device.
So they found that with a low-frequency scale,
only 14% of people picked a value of more than 2.5 hours.
On the high-frequency scale, 44% pick an option that was greater than 2.5 hours.
It's not surprising, if you look at the scale.
For the high-frequency scale,
this means 44% picked anything besides the first option.
So there seems to be some evidence that the basic response processes are similar
in Mobile Web and other modes, but there are features of mobile devices in
the way they are used that really could affect measurement or response quality.
Antoun investigated this in a study in which he had respondents answer
the same questions, either on a mobile device and
then on a conventional computer or in the opposite order.
First, conventional computer and then mobile device.
So he had a within-subjects comparison,
which strengthens the ability to make inferences and conclusions.
So there's some evidence that the response processes are quite similar,
the sort of basic response processes are quite similar in mobile, web and
other modes at least according to these replication findings by Peytchev and Hill.
More recently, Antoun has done a direct comparison between mobile and
conventional web surveys.
He used a crossover design in which the same respondents answered the same set of
questions in both modes, either they were first assigned to the mobile condition and
then answered the questions on a PC or vice versa.
So, this is actually gives him a much more powerful kind of comparison than some of
the other studies that we've talked about.
The findings were that there was no difference in satisficing.
Remember, these are mental shortcuts that survey respondents often take.
It's not that he didn't find any evidence of satisficing,
it's just that it was no different between the two types of web surveys.
There was one reversal.
He was using a measure of the length of open responses, the number of words or
characters as an indication of satisficing the ideas that it's easier for
respondents to answer fewer questions and he found that the open responses were
longer in mobile than PC and this is not what one might expect.
One might think that entering text on a small device is more complicated.
He doesn't really have an explanation, but that was a finding.
It's considered a reversal.
But for the other measures of satisficing such as straight lining,
giving the same answer to a number of questions and primacy effects,
there was really no difference between mobile and the PC Web.
No differences in disclosure of sensitive information.
So we earlier talked about the fact that in this study,
the mobile respondents were far more engaged with their
surroundings than when they answered the questions on a PC.
And yet, there was no effect on disclosure.
So the concern that being around the others,
being away from home would inhibit honest respondent
to sensitive questions seems not to be the case, but
Antoun did find a couple of pieces of evidence that
response quality is lower on mobile devices than on PCs.
And in particular, answers to questions about the respondent's age and
their birth year.
The reason he used he used these is he had true values in the sampling frame.
So, what he concluded was that it was the widgets that the respondents
used the data entry features that respondents used on the mobile
device that were responsible for lower accuracy.
They used a slider for their age and the date picker to indicate their birth year,
and the idea is that this requires considerable dexterity, and
it can be challenging on a small screen.
So his conclusion was that while most of the indications of data quality
that survey researchers use were no different between mobile and
web where screen size was an issue, there was a difference and
in the direction of apparently lower quality on mobile than PC data collection.
So, small screen seems to be sort of the culprit in the Antoun study.
This is actually consistent with some of the early evidence with Peytchev and Hill.
One finding in the literature they were unable to replicate was
a visual contrast effect that Couper and colleagues had shown him.
We talked about this when we talked about conventional web surveys and
measurement error in conventional web surveys, and you may recall the finding
was that people self-rated health was affected by an image on the screen or
actually previews screen of a person who's either quite healthy looking or
quite unhealthy looking.
Peytchev and Hill did essentially the same study and
they were unable to produce the same contrast effect in the sense that.
People rated their health higher when the image depicted someone who was quite
unhealthy than when it depicted someone who was quite healthy.
So they were unable to replicate this with similar images and
suggested number one, that this was a positive outcome,
because this is really a source of error that you would hope wouldn't occur.
But it's really kind of a null finding, which makes it hard to interpret.
Does this mean that there is never an effect of image on a mobile device,
on a small screen.
That's one limitation of this finding.
And also if the take-home message is that images are not noticed on a small screen,
that's not necessarily good news.
Because even designing for a small screen,
one would like to be able to judiciously present images.
Over all the studies we've now discussed,
response quality seems very much the same in mobile and
PC modes as long as the mobile response format is designed for a small screen.
One other aspect of Mobile Web surveys is that particularly in the early
days of Mobile Web responding, designers actually intended for
all responses to be entered through conventional devices not mobile devices.
So responses via mobile devices were considered unintended and
really unwanted, and designers did not optimize the design for mobile devices.
Bosnjak and colleagues compared these so-called unintended mobile
respondents to desktop, laptop participation into meta analysis
as they looked across a number of studies and summarized the findings.
In one study,
they found that the average participation rate -of unintended mobile users was 5.8%.
They found that younger respondents were generally more likely to
participate via mobile devices and
that males more than females were likely to participate on mobile devices.
The break-off rate was actually higher for mobile than desktop devices and
this could well be, because the design was not optimized for mobile devices.
That is the screens were not designed for a small display.
Despite the higher break-off rate for mobile than desktop devices, the number
of pages completed before break-off was actually higher on mobile than on desktop.
Not clear exactly why, but it could be that these respondents are quite
motivated to complete the questionnaire but eventually gave up.
In the other meta-analysis, Bosnjak and his colleagues found that about
8% of respondents used a mobile device, which was unintended at least once.
And that 2.1% of respondents participated via mobile device in at least seven
out of eight waves of this longitudinal study.
So really, in terms of prevalence, this unintended mode was relatively high.
They also found that older respondents and women were less likely to
use mobile devices, consistent with the finding in the other data analysis.
So to summarize our discussion of Mobile Web surveys, smartphone and
tablet coverage is far from the general population, but it is rapidly growing and
it is potentially an alternative to in-home internet access for at least
certain subgroups of the population who may have no other way to go online.
We talked about SMS as a contact mode and it does seem to be promising,
that is sending people a text message about an upcoming web survey seems
to be more effective than other modes such as email or a physical letter.
The other advantage of SMS is that it may potentially allow sampling of phone
numbers, which could overcome the number of the coverage issues that we spoke about
with web surveys.
The difficulty or the challenge for using SMS is that
it may be hampered by legal restrictions that unsolicited
SMS messages may be prohibited in certain countries.
And it may require that respondent's give prior consent to
being texted before a researcher tries to contact respondents that way.
When we looked at measurement, there was really very little evidence that mobile
and conventional web surveys differed in the quality of individual responses.
That is measurement error seemed to be about the same in the two
types of web surveys.
Finally, it may make sense to use Mobile Web for short location or
context based surveys, rather than for longer questionnaires as is really
the norm, or is more common with conventional web surveys.
The critical design challenge seems to be creating questions and other
measurement items that will work on small screens, as well as on larger screens.
So we'll turn next to a discussion of SMS or
texting as a data collection, or interview mode.
We've talked about SMS and texting as a way to contact potential responds, but
we haven't discussed it yet as a way to collect datas.
And it is actually being used widely in some parts of the world and
a number of investigations or evaluations of the method have taken place.
So, we'll turn to those next.
See you shortly.
Ознакомьтесь с нашим каталогом
Присоединяйтесь бесплатно и получайте персонализированные рекомендации, обновления и предложения.
Coursera делает лучшее в мире образование доступным каждому, предлагая онлайн-курсы от ведущих университетов и организаций.
© Coursera Inc., 2018 Все права защищены.
