This report summarizes analyses to assess the reliability and validity of the Veterans RAND 12-Item Health Survey (VR-12) metrics and patient-reported outcome performance measures (PMs). We evaluated differences among Medicare Advantage Organizations using five PMs based on Physical Component Summary and Mental Component Summary scores derived from the VR-12. Recommendations are made for future use of selected PMs.
- What is a reliable and valid patient-reported outcome performance measure in a target geriatric population with multiple chronic conditions?
The goals of patient care within a health care system are no longer focused solely on ensuring patients' survival or curing disease, but on optimizing their functional status and well-being. This is often best assessed through patient-reported outcome (PRO) performance measures (PMs), which allow health care systems and different provider groups to be compared. The purpose of this project was to develop PMs based on PROs, specifically Veterans RAND 12-Item Health Survey (VR-12) Physical Component Summary (PCS) and Mental Component Summary (MCS) scores, in individuals with multiple chronic conditions (MCC).
This report summarizes analyses to assess the reliability and validity of the VR-12 metrics and patient-reported outcome performance measures (PMs). We evaluated differences among Medicare Advantage Organizations using five PMs based on PCS and MCS scores derived from the VR-12. Recommendations are made for future use of selected PMs.
Performance Measures Based on the Mental Component Summary Score of the Veterans RAND 12-Item Health Survey Performed Better than Performance Measures Based on the Physical Component Summary in the Context of Differentiating Medicare Advantage Organizations
- For the five basic performance measures (PMs) applied to the entire Medicare Advantage Organization (MAO) samples, reliabilities for both physical component summary (PCS)– and mental component summary (MCS)–based PMs were low.
Modifying PM3 by Removing Sociodemographic Status Variables from the Risk-Adjustment Model Enhanced Intraclass Correlation Coefficients and Reliability
- The intraclass correlation coefficieint (ICC) and reliability of PM5 were enhanced by creating a threshold, based on quintiles of performance, by which lower- versus higher-performing MAOs could be compared.
- While the MCS-based PMs (PM3 and PM5) were improved, the reliability of the PCS-based PMs was not consistently improved to a meaningful degree by these modifications.
- Reliability was also improved by combining two or three cohorts, presumably because of the increased sample size. Combining the results across cohorts 13–15 gave a reliability of 0.67 for one of the modified MCS PMs. None of the PMs we examined achieved the usual threshold of 0.7 that is considered sufficient reliability for a PM, although some came close.
The Main Challenge to the Goals of This Project Is Related to the Characteristics of the Data
- The data reflect a nationally based system of care with considerable heterogeneity.
- An optimal database would include information to define individual providers or group practices within each of the MAOs.
- Future studies should further evaluate the psychometrics of these measures at the level of the group or the individual clinician practice.
- We predict that such an analysis would yield higher reliabilities for both PCS- and MCS-based PMs by reducing the noise or within-group variability and increase the signal for distinguishing performance among individual providers or provider groups.
Table of Contents
Construction of Patient-Reported Outcome Performance Measures
Define the PM and Conduct Analyses to Assess Reliability, Validity, and Threats to Validity
Develop and Refine Risk Adjustment of the Performance Measures Using Available Data and Modeling
Assess the Potential for the Performance Measure to Demonstrate Differences Between Ambulatory Practices and Between Health Care Systems
Conclusions and Recommendations
Methods of Calculating Reliability of Performance Measures
Association of Reliability to Size of MAO Enrollment in HOS Cohorts 13–15
Association of Reliability with the Number of Reported Conditions (≥ 3, ≥ 4, ≥ 5)
Association of Reliability with Particular Conditions as One of the MCCs (Depression, Arthritis, Angina/Coronary Disease/Heart Attack, Diabetes, Sciatica)
Association of Reliability with Age (65–75 and 75 and Older)
Association of Reliability with Length of Time Respondent Had Been in MAO Prior to Baseline Survey