Reliability of Utilization Measures for Primary Care Physician Profiling

Published in: Healthcare, v. 1, no. 1-2, June 2013, p. 22-29

Posted on on June 01, 2013

by Hao Yu, Ateev Mehrotra, John L. Adams

Read More

Access further information on this document at Healthcare

This article was published outside of RAND. The full text of the article can be found at the link above.

Research Questions

  1. Do common utilization measures reflect real variation in health costs among primary care physicians?
  2. Can these measures be used appropriately for physician profiling?

BACKGROUND: Given rising health care costs, there has been a renewed interest in using utilization measures to profile physicians. Despite the measures' common use, few studies have examined their reliability and whether they capture true differences among physicians. METHODS: A local health improvement organization in New York State used 2008–2010 claims data to create 11 utilization measures for feedback to primary care physicians (PCP). The sample consists of 2938 PCPs in 1546 practices who serve 853,187 patients. We used these data to measure reliability of these utilization measures using two methods (hierarchical model versus test–retest). For each PCP and each practice, we estimate each utilization measure's reliability, ranging from 0 to 1, with 0 indicating that all differences in utilization are due to random noise and 1 indicating that all differences are due to real variation among physicians. RESULTS: Reliability varies significantly across the measures. For 4 utilization measures (PCP visits, specialty visits, PCP lab tests (blood and urine), and PCP radiology and other tests), reliability was high (mean>0.85) at both the physician and the practice level. For the other 7 measures (professional therapeutic visits, emergency room visits, hospital admissions, readmissions, skilled nursing facility days, skilled home care visits, and custodial home care services), there was lower reliability indicating more substantial measurement error. CONCLUSIONS: The results illustrate that some utilization measures are suitable for PCP and practice profiling while caution should be used when using other utilization measures for efforts such as public reporting or pay-for-performance incentives.

Key Findings

  • Only four commonly used utilization measures reliably capture real differences in utilization among physicians.
  • Other measures commonly used for physician profiling have lower reliability and should be used with caution.


  • Given the substantial variation across utilization measures, organizations using them to profile physicians should rigorously evaluate the measures' reliability.

This report is part of the RAND Corporation External publication series. Many RAND studies are published in peer-reviewed scholarly journals, as chapters in commercial books, or as documents published by other organizations.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.