Research Question

  1. What is a reliable and valid patient-reported outcome performance measure in a target geriatric population with multiple chronic conditions?

The goals of patient care within a health care system are no longer focused solely on ensuring patients' survival or curing disease, but on optimizing their functional status and well-being. This is often best assessed through patient-reported outcome (PRO) performance measures (PMs), which allow health care systems and different provider groups to be compared. The purpose of this project was to develop PMs based on PROs, specifically Veterans RAND 12-Item Health Survey (VR-12) Physical Component Summary (PCS) and Mental Component Summary (MCS) scores, in individuals with multiple chronic conditions (MCC).

This report summarizes analyses to assess the reliability and validity of the VR-12 metrics and patient-reported outcome performance measures (PMs). We evaluated differences among Medicare Advantage Organizations using five PMs based on PCS and MCS scores derived from the VR-12. Recommendations are made for future use of selected PMs.

Key Findings

Performance Measures Based on the Mental Component Summary Score of the Veterans RAND 12-Item Health Survey Performed Better than Performance Measures Based on the Physical Component Summary in the Context of Differentiating Medicare Advantage Organizations

  • For the five basic performance measures (PMs) applied to the entire Medicare Advantage Organization (MAO) samples, reliabilities for both physical component summary (PCS)– and mental component summary (MCS)–based PMs were low.

Modifying PM3 by Removing Sociodemographic Status Variables from the Risk-Adjustment Model Enhanced Intraclass Correlation Coefficients and Reliability

  • The intraclass correlation coefficieint (ICC) and reliability of PM5 were enhanced by creating a threshold, based on quintiles of performance, by which lower- versus higher-performing MAOs could be compared.
  • While the MCS-based PMs (PM3 and PM5) were improved, the reliability of the PCS-based PMs was not consistently improved to a meaningful degree by these modifications.
  • Reliability was also improved by combining two or three cohorts, presumably because of the increased sample size. Combining the results across cohorts 13–15 gave a reliability of 0.67 for one of the modified MCS PMs. None of the PMs we examined achieved the usual threshold of 0.7 that is considered sufficient reliability for a PM, although some came close.

The Main Challenge to the Goals of This Project Is Related to the Characteristics of the Data

  • The data reflect a nationally based system of care with considerable heterogeneity.
  • An optimal database would include information to define individual providers or group practices within each of the MAOs.

Recommendations

  • Future studies should further evaluate the psychometrics of these measures at the level of the group or the individual clinician practice.
  • We predict that such an analysis would yield higher reliabilities for both PCS- and MCS-based PMs by reducing the noise or within-group variability and increase the signal for distinguishing performance among individual providers or provider groups.

Table of Contents

  • Chapter One

    Construction of Patient-Reported Outcome Performance Measures

  • Chapter Two

    Define the PM and Conduct Analyses to Assess Reliability, Validity, and Threats to Validity

  • Chapter Three

    Develop and Refine Risk Adjustment of the Performance Measures Using Available Data and Modeling

  • Chapter Four

    Assess the Potential for the Performance Measure to Demonstrate Differences Between Ambulatory Practices and Between Health Care Systems

  • Chapter Five

    Conclusions and Recommendations

  • Appendix A

    Methods of Calculating Reliability of Performance Measures

  • Appendix B

    Association of Reliability to Size of MAO Enrollment in HOS Cohorts 13–15

  • Appendix C

    Association of Reliability with the Number of Reported Conditions (≥ 3, ≥ 4, ≥ 5)

  • Appendix D

    Association of Reliability with Particular Conditions as One of the MCCs (Depression, Arthritis, Angina/Coronary Disease/Heart Attack, Diabetes, Sciatica)

  • Appendix E

    Association of Reliability with Age (65–75 and 75 and Older)

  • Appendix F

    Association of Reliability with Length of Time Respondent Had Been in MAO Prior to Baseline Survey

Research conducted by

The research described in this report was conducted by RAND Health.

This report is part of the RAND Corporation research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.