Methods Commonly Used to Create 'Report Cards' May Overestimate the Quality of Health Care

Methods commonly used to create medical “report cards” that grade the quality of care provided by doctors and hospitals may be resulting in grades that are too high, according to a new RAND Corporation study.

The study examined care provided to a group of vulnerable older patients. Researchers using information from claims filed to receive payment and other administrative records to assess the quality of medical care found that the patients received 83 percent of the recommended care. But when researchers graded the care using a broader set of standards after examining medical records — used less often for medical report cards — they found that same group of seniors received just 55 percent of the recommended care.

“Our findings suggest that examining administrative information alone does not provide an accurate picture of whether people are receiving appropriate medical care,” said the study's lead author, Dr. Catherine MacLean of RAND Health and the Greater Los Angeles VA Healthcare System. “It appears that report cards based on just administrative information are incomplete and may lead to grade inflation.”

The findings of the study, published in the February edition of Medical Care, add support to recommendations for creating computerized medical records. Such records could make it easier to incorporate detailed medical information into the routine creation of medical report cards.

“As efforts move forward to develop electronic medical records, we need to make sure they are constructed in a way that simplifies efforts to use them as the basis for health care quality assessments,” MacLean said.

While the RAND study focused on older patients, the findings raise questions about using administrative records for medical report cards among younger patients because several of the conditions studied — diabetes, depression and high blood pressure — also commonly occur among younger people.

Public agencies and private groups are creating report cards to help consumers choose doctors, hospitals and insurance plans. In addition, performance on such report cards may affect the fees paid to doctors and hospitals. Under so-called "pay for performance" plans, health providers that score higher on report cards may receive higher fees or bonuses, or their patients may pay lower copayment amounts.

Efforts to develop report cards have relied upon administrative records collected as a part of insurance payment systems because the information generally is computerized, making it easier to analyze how well large numbers of patients are treated.

By contrast, most medical records are handwritten notes made by doctors, nurses and other health providers. Using these records to evaluate medical quality requires that patients give permission for copies to be collected from various sources and reviewed by experts so the information can be put into standardized formats that allow many patients' care to be compared.

The findings are from a project called Assessing the Care of Vulnerable Elders (ACOVE), an effort to examine the quality of health care provided to vulnerable older Americans who live independently. The project is a partnership between RAND Health and Pfizer Inc.

The study examined the medical care given over a 13-month period to 399 patients age 65 or older who were at risk of declining health and were enrolled in two managed-care health plans in different regions of the nation.

“These are the people who use the most health care resources and the people for whom we have previously shown that health quality makes a big difference,” MacLean said.

Researchers examined performance on 182 measures of health care quality received by the elderly patients for 22 common medical conditions, including pneumonia, heart disease, malnutrition and urinary incontinence. For example, one measure was whether patients with diabetes received an annual blood test to measure the control of their blood sugar.

About 20 percent of the quality measures could be examined using either administrative information or medical records. Among this set of measures, both sets of records agreed that patients received 83 to 84 percent of the recommended medical care.

But when the assessment was expanded to include the other 80 percent of measures that could only be checked through a thorough review of medical records, the study found patients received just 55 percent of the recommend medical care.

In addition, administrative records could not provide any information about quality measures for five conditions important to the elderly — end-of-life care, falls, malnutrition, pressure ulcers and urinary incontinence.

The study is titled “Comparison of Administrative Data and Medical Records to Measure the Quality of Medical Care Provided to Vulnerable Older Patients.”

Other authors of the report are Rachel Louie, Caren J. Kamberg, Carol P. Roth, John Adams and Dr. David H. Solomon, all of RAND; Dr. Paul Shekelle and Dr. Debra M. Saliba, of both RAND and the Greater Los Angeles VA Healthcare System; Dr. Takahiro Higashi of UCLA's David Geffen School of Medicine; Dr. John T. Chang, Dr. Roy T. Young and Dr. Neil S. Wenger, all of RAND and UCLA's David Geffen School of Medicine.

RAND Health is the nation's largest independent health policy research organization, with a broad research portfolio that focuses on health care quality, costs, and delivery, among other topics.

Pfizer Inc discovers, develops, manufactures and markets leading prescription medicines, for humans and animals, and many of the world's best-known consumer brands.