Physician Cost Profiling -- Reliability and Risk of Misclassification: Detailed Methodology and Sensitivity Analyses
Feb 26, 2010
Passage of health care reform legislation by the 111th Congress represents major progress in reducing the number of the uninsured. Now the focus of policy discussion is likely to turn quickly from issues of coverage to the challenge of controlling health care costs.
Purchasers have been experimenting for some time with different approaches to reining in costs (consumer cost sharing, managed care, financial incentives for physicians). Most of these approaches have either met with resistance from consumers (e.g., managed care restricting choice) or have been ineffective (e.g., regulatory strategies).
The latest approach is to focus attention on the relative costs of individual physicians and to give consumers incentives to seek care from physicians with relatively lower spending patterns. It makes sense to focus on physicians because they drive much of health care spending through the decisions they make (e.g., which tests to order, what drugs to prescribe, which referrals to make, and what procedures to perform).
Purchasers have developed methods for creating overall assessments of practice patterns called cost profiles. A cost profile is a single number that places a physician on a relative scale of spending. Purchasers use cost profiles to identify physicians who are high cost or low cost (similar in concept to the number of dollar signs indicating the expense one might encounter at a particular restaurant).
Creating the profiles requires many technical decisions. Until recently, those decisions were largely a black box—different organizations made different choices. Doctors were largely not informed about how their cost profiles were created. Some doctors found that two health plans might give them two different cost profiles (e.g., one plan said they were high cost, while another found the opposite). Understanding the process of cost profiling is especially important now because under the new health care reform legislation, Medicare will be providing physician cost profiles (called relative resource reports) by 2012.
To open the black box of physician cost profiling, RAND analysts conducted a series of studies to examine the different technical decisions necessary for creating cost profiles and the utility of cost profiles for helping consumers select more "efficient" doctors. The RAND team's goal was to understand whether the decisions made in creating the profile affected whether a physician is labeled high cost or low cost.
To form the empirical basis for this work, the researchers analyzed aggregated claims data on 1.1 million adults age 18–65 continuously enrolled in four Massachusetts commercial health plans in 2004 and 2005. The study sample included more than 13,700 physicians. The team used methods for constructing physician cost profiles and assigning physicians to cost categories that largely mirrored methods commonly used by health plans.
|Number of services
|Average price per service ($)
|Office visits with primary care physician
|Treadmill stress test to see if coronary artery disease is stable
|One-year supply of medication for coronary artery disease
|Total observed costs of episode
Over the course of a year a patient with coronary artery disease visited her physician four times, had a treadmill test, obtained a year's supply of a necessary drug, and had a blood test to determine her cholesterol and triglyceride levels. She also had a consultation with a cardiologist.
The costs of the services in this example are fictional. In the analyses, the cost of each service was the average charge allowed by the relevant health plans—in the case of RAND's study, the four Massachusetts health plans.
Creating a physician cost profile involves multiple steps. These include deciding which care will be included in a cost profile (e.g., the inclusion or exclusion of preventive care), determining which costs are assigned to each unit of care, deciding which physician is responsible for care, identifying the peers to whom a physician will be compared, creating the algebra for constructing cost profiles, and deciding how to place a physician in a high-cost or low-cost category. This research highlight summarizes research on a few of these decisions.
The initial decision is whether the allocation of dollars spent on health care should be by patient—thus assigning all of the patient's costs to a single physician—or by episodes of care. Episodes of care are the clinically related services delivered to a patient with a specific condition over a certain period of time; currently, health plans commonly use commercial software to group a patient's claims into an episode of care (see Table 1 for an example).
Another key step is deciding how to assign accountability for costs. Patients often see many doctors. Which doctor should be accountable for the costs of care? One option is to assign costs to the physician who had the most visits, either from the patient or during an episode of care. Another approach would be to assign costs to the physician who accounted for largest portion of spending.
Health plans use the results of such decisions (referred to as attribution rules) to assign costs of care to a physician and categorize physicians as low, average, or high cost. But there are many permutations and combinations of the attribution rules. Does a given physician end up in the same cost category no matter which attribution rules are used? And once a cost profile is created, how reliable is it?
Most health plans assign physicians to one of two or three cost categories (e.g., low, average, or high cost). To gauge how the choice of attribution rules might affect the category to which a physician is assigned, RAND researchers created 12 attribution rules reflecting the current range of rules being used by insurers and calculated what fraction of physicians was assigned to a different cost category by each of 11 attribution rules, compared with the default rule. (The default rule reflected a rule currently in wide use among health plans. It uses episodes as the unit of analysis and assigns costs to the physician who generated the largest percentage of costs.)
RAND's assessment showed that the choice of attribution rules strongly affected a physician's cost category. Specifically, 17 to 61 percent of physicians would be assigned to a different cost category if a rule other than the default attribution rule were used.
A related issue is the reliability of cost profiles. Once a cost profile has been created, how accurately does it reflect a physician's actual economic performance? RAND researchers used the Massachusetts data described above to answer this question.
For each physician in the study, RAND analysts constructed a summary cost profile score indicating what the costs were for the episodes of care assigned to a given physician, relative to the costs of other physicians in the same specialty who were treating patients with similar conditions and similar comorbidities. Then the analysts assessed the reliability of the profile—that is, the extent to which variation across cost profile scores reflected real variations in physicians' economic performance or simply error in how the cost profile score was constructed. Reliability is expressed on a scale from 0 to 100 percent, where 0 percent means that differences in cost profile scores result solely from measurement error, and 100 percent means that all of the differences in scores reflect actual variations in physician resource use.
RAND found that the median reliability of physician cost profiles varied widely, especially across specialties, ranging from 5 percent for vascular surgery to 97 percent for dermatology (see Figure 1). Overall, 59 percent of physicians had cost profile reliability scores below 70 percent, a commonly used reliability threshold.
Purchasers frequently require a minimum sample size before they profile individual physicians. For example, they may not profile physicians to whom fewer than 30 episodes of care have been assigned.
But the analysts found that minimum cutoffs are a relatively poor proxy for reliability. Figure 2 charts the relationship between sample size and reliability; each dot is a physician. The two circles highlight two doctors, each of whom was assigned more than 250 episodes. But one physician has a cost profile with a high reliability (almost 0.95); the other has a low reliability profile.
The RAND team used the estimates of reliability to measure the probability that a physician would be inaccurately categorized in a two-tiered classification system. In this system, physicians whose cost profiles were in the lower 25 percent of all profiles were designated as lower cost; all others were considered not lower cost.
The analysts found that many physicians were misclassified. In some cases, physicians who were actually lower cost were categorized as not lower cost; in other cases, the opposite was true (see Table 2). Overall, about 22 percent of physicians were assigned to the wrong cost category.
|Physicians Misclassified as Lower Cost (%)
|Physicians Misclassified as Not Lower Cost (%)
|Overall Misclassification Rate (%)
These substantial rates of misclassification are cause for concern. If more than 40 percent of physicians classified as lower cost are not actually lower cost, then the ability of insurance plans to control costs by channeling patients to lower-cost physicians could be severely hampered. Patients might also be disappointed if they were to change physicians expecting to receive lower-cost care but then not realize any savings.
RAND researchers have examined multiple dimensions of physician cost profiling (see the gray box below). All of the work was based on data from non-Medicare beneficiaries in one state. Nevertheless, the work strongly suggests that current methods of physician cost profiling are not ready for prime time. However, the need to curtail rising health care costs is going to become more salient in the policy debate, not less. So current cost profiling approaches need to be improved, or new approaches need to be developed. In the interim, insurers and others need to be transparent about how they create and use physician cost profiles.