Cover: Establishing Limits for Supplemental Items on a Standardized National Survey

Establishing Limits for Supplemental Items on a Standardized National Survey

Published in: Public Opinion Quarterly, 2016

Posted on Oct 20, 2016

by Megan K. Beckett, Marc N. Elliott, Sarah J. Gaillot, Ann C. Haas, Jacob W. Dembosky, Laura Giordano, Julie A. Brown

Given the high costs of collecting survey data, adding supplemental questions to large, ongoing survey data collection efforts potentially offers a cost-effective way to collect additional information customized to the specific needs of users of a general survey. The Medicare Advantage Consumer Assessment of Healthcare Providers and Systems (MA CAHPS) Surveys use mail with telephone follow-up to survey Medicare beneficiaries about their experiences with health plans. These surveys are administered by trained survey vendors, who were allowed to add any number of supplementary survey items to the core survey in 2012, as long as the items met content requirements. Vendors and health plans create these supplemental items for internal quality improvement efforts. Because the results of the core items from the surveys are publicly reported and used to calculate Medicare Quality Bonus Payments to health plans under the Affordable Care Act, concern arose that an excessive number of supplementary items might adversely affect response rates and the reliability of plan-level estimates. To estimate the effect of supplemental items on response rates, we used logistic regression, controlling for survey vendor and characteristics of beneficiaries and plans. Response rates were lower in both telephone and mail modes with more supplemental items (p < 0.001). The use of 12 supplemental items was associated with response rates 2.5 percentage points lower, compared with surveys with no supplemental items. These results were used to develop new guidelines suggesting that supplemental items be limited to a maximum of 12.

This report is part of the RAND external publication series. Many RAND studies are published in peer-reviewed scholarly journals, as chapters in commercial books, or as documents published by other organizations.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.