This report summarizes analyses to assess the reliability and validity of the Veterans RAND 12-Item Health Survey (VR-12) metrics and patient-reported outcome performance measures (PMs). We evaluated differences among Medicare Advantage Organizations using five PMs based on Physical Component Summary and Mental Component Summary scores derived from the VR-12. Recommendations are made for future use of selected PMs.

Outcome Performance Measure Development for Persons with Multiple Chronic Conditions

Lewis E. Kazis, William H. Rogers, James Rothendler, Shirley Qian, Alfredo Selim, Maria Orlando Edelen, Brian D. Stucky, Adam J. Rose, Emily Butcher

ResearchPublished Apr 12, 2017

The goals of patient care within a health care system are no longer focused solely on ensuring patients' survival or curing disease, but on optimizing their functional status and well-being. This is often best assessed through patient-reported outcome (PRO) performance measures (PMs), which allow health care systems and different provider groups to be compared. The purpose of this project was to develop PMs based on PROs, specifically Veterans RAND 12-Item Health Survey (VR-12) Physical Component Summary (PCS) and Mental Component Summary (MCS) scores, in individuals with multiple chronic conditions (MCC).

This report summarizes analyses to assess the reliability and validity of the VR-12 metrics and patient-reported outcome performance measures (PMs). We evaluated differences among Medicare Advantage Organizations using five PMs based on PCS and MCS scores derived from the VR-12. Recommendations are made for future use of selected PMs.

Key Findings

Performance Measures Based on the Mental Component Summary Score of the Veterans RAND 12-Item Health Survey Performed Better than Performance Measures Based on the Physical Component Summary in the Context of Differentiating Medicare Advantage Organizations

  • For the five basic performance measures (PMs) applied to the entire Medicare Advantage Organization (MAO) samples, reliabilities for both physical component summary (PCS)– and mental component summary (MCS)–based PMs were low.

Modifying PM3 by Removing Sociodemographic Status Variables from the Risk-Adjustment Model Enhanced Intraclass Correlation Coefficients and Reliability

  • The intraclass correlation coefficieint (ICC) and reliability of PM5 were enhanced by creating a threshold, based on quintiles of performance, by which lower- versus higher-performing MAOs could be compared.
  • While the MCS-based PMs (PM3 and PM5) were improved, the reliability of the PCS-based PMs was not consistently improved to a meaningful degree by these modifications.
  • Reliability was also improved by combining two or three cohorts, presumably because of the increased sample size. Combining the results across cohorts 13–15 gave a reliability of 0.67 for one of the modified MCS PMs. None of the PMs we examined achieved the usual threshold of 0.7 that is considered sufficient reliability for a PM, although some came close.

The Main Challenge to the Goals of This Project Is Related to the Characteristics of the Data

  • The data reflect a nationally based system of care with considerable heterogeneity.
  • An optimal database would include information to define individual providers or group practices within each of the MAOs.

Recommendations

  • Future studies should further evaluate the psychometrics of these measures at the level of the group or the individual clinician practice.
  • We predict that such an analysis would yield higher reliabilities for both PCS- and MCS-based PMs by reducing the noise or within-group variability and increase the signal for distinguishing performance among individual providers or provider groups.

Topics

Document Details

Citation

RAND Style Manual
Kazis, Lewis E., William H. Rogers, James Rothendler, Shirley Qian, Alfredo Selim, Maria Orlando Edelen, Brian D. Stucky, Adam J. Rose, and Emily Butcher, Outcome Performance Measure Development for Persons with Multiple Chronic Conditions, RAND Corporation, RR-1844-NIH, 2017. As of October 11, 2024: https://www.rand.org/pubs/research_reports/RR1844.html
Chicago Manual of Style
Kazis, Lewis E., William H. Rogers, James Rothendler, Shirley Qian, Alfredo Selim, Maria Orlando Edelen, Brian D. Stucky, Adam J. Rose, and Emily Butcher, Outcome Performance Measure Development for Persons with Multiple Chronic Conditions. Santa Monica, CA: RAND Corporation, 2017. https://www.rand.org/pubs/research_reports/RR1844.html.
BibTeX RIS

The research described in this report was conducted by RAND Health.

This publication is part of the RAND research report series. Research reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND research reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.