Establishing Limits for Supplemental Items on a Standardized National Survey
Published in: Public Opinion Quarterly, 2016
Posted on RAND.org on October 20, 2016
Given the high costs of collecting survey data, adding supplemental questions to large, ongoing survey data collection efforts potentially offers a cost-effective way to collect additional information customized to the specific needs of users of a general survey. The Medicare Advantage Consumer Assessment of Healthcare Providers and Systems (MA CAHPS) Surveys use mail with telephone follow-up to survey Medicare beneficiaries about their experiences with health plans. These surveys are administered by trained survey vendors, who were allowed to add any number of supplementary survey items to the core survey in 2012, as long as the items met content requirements. Vendors and health plans create these supplemental items for internal quality improvement efforts. Because the results of the core items from the surveys are publicly reported and used to calculate Medicare Quality Bonus Payments to health plans under the Affordable Care Act, concern arose that an excessive number of supplementary items might adversely affect response rates and the reliability of plan-level estimates. To estimate the effect of supplemental items on response rates, we used logistic regression, controlling for survey vendor and characteristics of beneficiaries and plans. Response rates were lower in both telephone and mail modes with more supplemental items (p < 0.001). The use of 12 supplemental items was associated with response rates 2.5 percentage points lower, compared with surveys with no supplemental items. These results were used to develop new guidelines suggesting that supplemental items be limited to a maximum of 12.