Loss Function-Based Evaluation of Physician Report Cards

Published in: Health Services and Outcomes Research Methodology [Epub February 2018]. doi:10.1007/s10742-018-0179-2

Posted on RAND.org on April 17, 2018

by Fernando Hoces de la Guardia, Jessica Hwang, John L. Adams, Susan M. Paddock

Read More

Access further information on this document at Springer US

This article was published outside of RAND. The full text of the article can be found at the link above.

Report cards classifying physicians into performance tiers are central to health care quality improvement initiatives. Misclassification is a concern since physicians often have small patient panels for standard performance measures. Given that report cards are used for different purposes by different stakeholders, we specify loss functions and evaluate the potential cost of misclassification for physician report card designs. Monte Carlo simulation to explore misclassification risk and cost for four illustrative physician report card designs and three loss functions representing overall misclassification, patient, and pay-for-performance program perspectives. True physician performance is simulated under a beta-binomial model with parameters yielding simulated true scores resembling previously reported estimates. Misclassification risk varied across report card designs. Overall misclassification risk increased with the number of performance tiers for our simulated scenarios. However, the relationship between misclassification cost and number of tiers was inconsistent across the loss functions. The report card with the lowest misclassification cost varied across stakeholders. Within stakeholder, the costs of a two-tier report card with a high or low hurdle (25th and 75th percentile, respectively) varied. Loss functions and report card designs are illustrative and not intended to exhaustively catalog all possibilities. Little guidance exists on misclassification costs from the patient perspective. Misclassification cost depends on how performance information will be used and by whom. Selecting the lowest-cost design for a given stakeholder could maximize the usefulness of physician performance data. Misclassification cost could guide report card design, improving the usefulness of a report card for one stakeholder without disadvantaging others.

This report is part of the RAND Corporation external publication series. Many RAND studies are published in peer-reviewed scholarly journals, as chapters in commercial books, or as documents published by other organizations.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.