An Item Response Theory Approach to Estimating Survey Mode Effects

Analysis of Data from a Randomized Mode Experiment

Published in: Journal of Survey Statistics and Methodology, Volume 5, Issue 2 (June 2017), pages 233-253. doi: 10.1093/jssam/smw033

Posted on RAND.org on May 31, 2017

by Louis T. Mariano, Marc N. Elliott

Read More

Access further information on this document at Journal of Survey Statistics and Methodology

This article was published outside of RAND. The full text of the article can be found at the link above.

When a survey is offered in more than one mode of administration, the potential for differences in the probability of selecting a response category attributable to the survey administration mode may threaten the cross-mode exchangeability of responses or comparability of results. We demonstrate the utility of item response theory (IRT) in quantifying the presence of mode effects, providing insight into the nature of the effects and adjusting cross-mode results. Such IRT applications are of interest when the survey instrument informs an underlying latent trait. We present a Bayesian hierarchical IRT model that can accommodate multiple modes of survey administration and provide cluster-level parameter estimates of the latent trait when observed groupings of respondents are of interest. We illustrate the model with data from a randomized survey mode experiment in which responding subjects within each of forty-five evaluated institutions were randomly assigned to one of four response modes: mail, telephone, interactive voice response (IVR), and a mixed mode of mail with telephone follow-up. Results indicate instrument-wide survey mode effects that differ across ordinal response categories and the underutilization of an interior response category in certain modes.

This report is part of the RAND Corporation External publication series. Many RAND studies are published in peer-reviewed scholarly journals, as chapters in commercial books, or as documents published by other organizations.

Our mission to help improve policy and decisionmaking through research and analysis is enabled through our core values of quality and objectivity and our unwavering commitment to the highest level of integrity and ethical behavior. To help ensure our research and analysis are rigorous, objective, and nonpartisan, we subject our research publications to a robust and exacting quality-assurance process; avoid both the appearance and reality of financial and other conflicts of interest through staff training, project screening, and a policy of mandatory disclosure; and pursue transparency in our research engagements through our commitment to the open publication of our research findings and recommendations, disclosure of the source of funding of published research, and policies to ensure intellectual independence. For more information, visit www.rand.org/about/principles.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.