This technical report explains the relationship between reliability measurement and misclassification for physician quality and cost measures in health care. It provides details and a practical method to calculate reliability and misclassification from the data typically available to health plans. This report builds on other RAND work on reliability and misclassification and has two main goals. First, it can serve as a tutorial for measuring reliability and misclassification. Second, it will describe the likelihood of misclassification in a situation not addressed in our prior work in which physicians are categorized using statistical testing. For any newly proposed system, the methods presented here should enable an evaluator to calculate the reliabilities and, consequently, the misclassification probabilities. It is our hope that knowing these misclassification probabilities will increase transparency about profiling methods and stimulate an informed debate about the costs and benefits of alternative profiling systems. The appendixes provide more technical detail on how to measure reliability with related program code as well as a set of lookup tables that can be used to obtain the rate of misclassification associated with a reliability estimate under various scenarios.
Adams, John L., Ateev Mehrotra, and Elizabeth A. McGlynn, Estimating Reliability and Misclassification in Physician Profiling. Santa Monica, CA: RAND Corporation, 2010. https://www.rand.org/pubs/technical_reports/TR863.html.
Adams, John L., Ateev Mehrotra, and Elizabeth A. McGlynn, Estimating Reliability and Misclassification in Physician Profiling, RAND Corporation, TR-863-MMS, 2010. As of November 11, 2022: https://www.rand.org/pubs/technical_reports/TR863.html