Modern Psychometric Methods for Estimating Physician Performance on the Clinician and Group CAHPS® Survey

Modern psychometric methods for scoring the Clinician & Group Consumer Assessment of Healthcare Providers and Systems (CG-CAHPS®) instrument can improve the precision of patient scores. The extent to which these methods can improve the reliable estimation and comparison of individual physician performance, however, remains unclear. Using CG-CAHPS® data from 12,244 unique patients of 448 primary care physicians in southern California, four methods were used to calculate composite scores: (1) standard scoring, (2) a single factor confirmatory factor analysis model, (3) a bifactor model, and (4) a correlated factor model. We extracted factor scores for physicians from each model and adjusted the scores for respondent characteristics, including age, education, self-rated physical health, and race/ethnicity. Physician-level reliability and physician rankings were examined across the four methods. The bifactor and correlated factor models achieved the best fit for the core CG-CAHPS® questions from the three core composite measures. Compared to standard adjusted scoring, the bifactor model scores resulted in a 25 % reduction in required sample sizes per physician. The correlation of physician rankings between scoring methods ranged from 0.58 to 0.86. The discordance of physician rankings across scoring methods was most pronounced in the middle of the performance distribution. Using modern psychometric methods to score physician performance on the core CG-CAHPS® questions may improve the reliability of physician performance estimates on patient experience measures, thereby reducing the required respondent sample sizes per physician compared to standard scoring. To assess the predictive validity of the CG-CAHPS® scores generated by modern psychometric methods, future research should examine the relative association of different scoring methods and important patient-centered outcomes of care.