Measuring the Quality of Cancer Care

Breast-cancer patients in the United States have a 1-in-7 chance of missing some of the care they should be receiving, and colorectal cancer patients have a 1-in-5 chance of missing recommended care, according to a newly released RAND Health study that developed a prototype system for monitoring the quality of cancer care received by cancer patients. The good news is that they stand a good chance of receiving high-quality care, far better than the odds for the more long-term—and silent—chronic conditions, such as hypertension and diabetes, according to another recent RAND Health study. However, these odds represent averages: The chance of receiving some types of desirable care can be as low as 36 percent.

Why the Need for a Quality Monitoring System?

The Institute of Medicine (IOM), the medical research arm of the National Academy of Sciences, issued a report in 1999 that concluded that many cancer patients in the United States were not receiving state-of-the-art care. This finding was surprising to many oncologists because of the tremendous improvement in treatment knowledge that has resulted from clinical trials, the gold standard of research studies. However, research on quality has demonstrated that knowledge does not automatically translate to optimal care. The IOM raised concerns about the underuse and overuse of care and the fact that outcomes of treatment for the general population are not directly comparable to outcomes of clinical trials. It recommended a number of steps, including development of a quality monitoring system that would take advantage of recent improvements in our ability to measure quality.

In response to this challenge, researchers at RAND Health, in collaboration with the Harvard Medical School, relevant professional organizations, and a task force of expert advisers, have developed a cancer-care quality monitoring system. Called the National Initiative for Cancer Care Quality (NICCQ), this prototypical system measures and monitors care quality for two common cancers—breast cancer and colorectal cancer—and has been tested in five metropolitan areas.

Developing the Quality Monitoring System

The development of the NICCQ was guided by two requirements: the need to measure and report on the quality of cancer care as accurately as possible for the lowest achievable cost, and results that support and inform quality improvement efforts.

Based on discussions with clinical experts, professional societies, and other interested persons, the researchers deemed four key features to be critical in developing the NICCQ: a representative patient sample, assurance of privacy, valid measures of care quality, and multiple data sources.

Selecting Patients

For the pilot test of the monitoring system, a carefully designed selection protocol was needed to ensure that the patients whose care would be examined actually represent the population of patients with the same cancer diagnosis. Unlike clinical trials, which select participants who meet narrowly defined inclusion criteria (such as the absence of other illnesses, also known as comorbid conditions; membership in a particular age and/or ethnic group; or likelihood of survival), an accurate quality measurement system must capture a cross section of all patients with the condition of interest.

Also important was the source of participants. Each potential source—hospital cancer registries, insurance rosters, large-scale nationwide health surveys—offered both advantages and disadvantages. The researchers settled on the American College of Surgeons' National Cancer Database, because it obtains information on some 70 percent of all cancers nationwide, a larger proportion than can be obtained from alternative sources. A final consideration was whether to select only newly diagnosed patients or patients in all stages of cancer. The NICCQ decided to include only newly diagnosed patients, because the clinical care processes that produce the greatest effect on outcome often take place shortly after diagnosis.

Protecting Privacy

Protecting privacy and confidentiality is critical for cancer patients. For example, inappropriate disclosure of their diagnosis could compromise their employability. The NICCQ researchers found that a properly designed and conducted quality monitoring system poses little risk to patients.

Defining Valid Measures of Cancer-Care Quality

It was imperative that the NICCQ include rigorously developed, validated measures of the quality of cancer care that had been demonstrated to improve outcomes, such as patients' likelihood of surviving, remaining disease free, or otherwise having a good quality of life. Choosing such measures for cancer-care quality presented several challenges. For example, an issue that always arises when developing quality measures is whether to measure the performance of care (process measures) or the results (outcome measures). Although it is easy to assume that the only true measure of quality is the outcome—improved survival or quality-adjusted life years—individual outcomes are very much affected by factors other than care quality, such as comorbid conditions. Thus, the researchers chose a combination of process and outcome measures, based on scientific evidence or, when such evidence was lacking, on expert consensus.

Another challenge, given rapidly evolving knowledge, is the need to ensure that measures are up to date—that is, that they reflect the latest treatment guidelines. Yet, at the same time, guidelines used to assess care provided in the past must reflect the standards of that time.

The project team developed measures of care processes within a set of five domains that represent important aspects of cancer care. (Table 1 shows these domains, along with an example of a measure from each category.) The measures specify events or situations that should cause a clinician to recommend a particular type of clinical care—a diagnostic test, treatment, or other intervention—for a particular patient, given the available evidence or expert consensus. In addition, adherence was measured for eight components of care that cut across all five domains: (1) testing; (2) pathology; (3) documentation; (4) referral; (5) timing; (6) receipt of treatment; (7) technical quality; and (8) respect for patient preferences.

Collecting Data from Multiple Sources

The information needed for a comprehensive assessment of a cancer patient's care often must be obtained from a number of different sources, such as the patient's hospital records, the records of several different individual physicians or outpatient clinics, tumor registries, and even the patients themselves, because no single source has all the information relevant to a patient's treatment. Thus, a third requirement for the NICCQ was a comprehensive and ongoing data-collection protocol that relies on several sources of available data: registries, medical records, and patient surveys. Patient surveys are particularly important for capturing information about current health status, care experiences, involvement in decisionmaking, quality of life, functional outcomes, and coordination of care—information that is rarely available from medical records.

The NICCQ collected data in two ways: via a patient survey and via review of medical records. Administration of the patient survey first addressed a number of logistical challenges, such as gaining patient consent and permission for record release. Further, in addition to providing information about their experiences and outcomes of care, patients were asked to provide the names and addresses of doctors, hospitals, and other providers involved in their care, which proved to be the most effective way to identify relevant medical records. Researchers accessed the medical records of the doctors named by the patients, as well as records of any other doctors named in those initial records. Data obtained from medical records included detailed information about tumor characteristics, staging, referrals and decisionmaking, initial treatment as well as any additional treatments, and comorbid conditions from 3 months before diagnosis through 12 months after the diagnosis.

For a patient to be eligible for a particular measure, the patient's data file had to include the records for the specialist generally responsible for providing the care being measured and the type of document that would normally show the type of information in question. For example, the measure "IF a patient is treated with chemotherapy, THEN body-surface area should be documented" would apply only to a patient whose medical record included the records of the medical oncologist and the chemotherapy flowsheet.

Quality Is Good but Highly Variable

The Patients and the Measures

Using hospital registries, the researchers recruited 400 patients with either breast or colorectal cancer from each of five metropolitan areas—Atlanta, Cleveland, Houston, Kansas City, and Los Angeles. Participation was limited to those who were between 21 and 80 years of age at the time of diagnosis, were alive at the time of contact (approximately four years after diagnosis), and were able to speak and write English.

For breast cancer, the researchers defined 36 quality measures. On average, breast-cancer patients received 86 percent of recommended care. Adherence to recommended care within the five clinical care domains varied somewhat, ranging from 73 percent (management of treatment toxicity) to 94 percent (post-treatment surveillance). However, adherence for different components of care ranged much more widely, from 13 percent (referral) to 60 percent (documentation) to 96 percent (testing) and 97 percent (receipt of treatment).

For colorectal cancer, the researchers defined 25 quality measures. With respect to these measures, patients received an average of 78 percent of recommended care. Adherence to recommended care within the five care domains ranged from 50 percent (post-treatment surveillance) to 93 percent (surgery). Adherence across the different components of care varied somewhat less, ranging from 57 percent (respect for patient preferences) and 59 percent (referral) to 93 percent (pathology).

Variation in Quality Across Metropolitan Areas

To find out whether quality varied by region, the researchers compared adherence to recommended care among the five metropolitan areas.

The average overall adherence for each of the metropolitan areas ranged from 82 to 87 percent, and no one region consistently provided better care across all domains. When specific types of care were compared among the metropolitan areas, the greatest variation was seen for breast-cancer surgery, followed by diagnostic evaluation and adjuvant therapy for breast cancer.

Among the 36 quality measures for breast-cancer care, adherence was less than 85 percent for half: Regional variation was significant for seven of those measures. For example, among patients treated with chemotherapy, the planned dose was consistent with published guidelines for only 61 percent of the patients in the study, a figure that ranged from 32 to 75 percent across metropolitan areas. Interestingly, six of the 18 measures with less than 85 percent adherence pertained to chart documentation of key clinical factors that are believed to be essential for providing appropriate cancer care.

Among the 25 quality measures for colorectal cancer, adherence was less than 85 percent for 14, but only one showed significant regional variation (radiation therapy for stage II or III rectal cancer). Only 66 percent of all stage II and III patients received recommended radiation therapy, and this figure varied from 58 to 92 percent across metropolitan areas.

What Do the Findings Mean for the Quality of Cancer Care?

The current quality of care for breast and colorectal cancers clearly contains some good news and some bad news. The good news is that a large proportion of patients is receiving recommended care. Adherence actually approached 100 percent for several measures, showing that excellent quality of care is possible. However, the bad news is that adherence was less than 85 percent for nearly half the measures, and for some measures, the quality varied substantially from one metropolitan area to another.

Cancer-care quality appears better than care for other chronic illnesses, according to several of RAND Health's recent studies, which suggest that people receive recommended care, on average, only about half the time. The findings also suggest that the quality of cancer care is better than earlier studies would indicate. There may be several possible reasons.

First, in contrast to prior studies, the NICCQ study used explicit quality measures with strictly defined eligibility criteria. Simply measuring utilization rates probably underestimates the appropriate use of care, because it fails to take into account how many patients are actually eligible for the process in question.

Second, few other studies have used the comprehensive data-collection methods used in the NICCQ. If many of the records used for the study were actually missing data (such as documentation of a treatment provided), care might appear to be of poorer quality than it actually was, since treatments might have been administered but not documented in the records used for data collection, a particularly likely possibility for care delivered in outpatient settings.

A third, related problem is that the evidence necessary for determining eligibility for a particular element of care might not have been in the records. However, the NICCQ went to great lengths to ensure complete records, which included contacting over 4,000 physicians.

If cancer-care quality is indeed higher than that for other illnesses, why? Perhaps the urgency of a new cancer diagnosis focuses providers' and patients' attention on treatment in a way that chronic diseases (especially "silent" ones, such as hypertension and diabetes) do not. Another possibility is that providers of cancer care might have greater confidence in the effectiveness of the care they provide than do physicians in other specialties. Still, the findings do demonstrate many areas in which care quality could be improved, such as the use of appropriate doses, appropriate timing of treatment initiation, and accurate documentation of the information needed for treatment and care.

The evaluation of the NICCQ was not without its limitations. First, the assessment was limited to two types of cancer and to certain stages of those cancers. Care quality might be different for other types of cancer or for more- or less-advanced breast and colorectal cancer. Second, the selected measures reflect both scientific evidence and expert consensus. Differences in the strength of the evidence as perceived by physicians could affect performance of particular measures. Third, the study excluded patients who could not read English or who had died shortly after diagnosis, two groups who might have received lower-quality care. Fourth, nonparticipation of particular hospitals or patients could also limit the generalizability of the findings.

Lessons Learned

In spite of its limitations, the NICCQ provided a snapshot of the quality of cancer care across the United States, as well as some valuable lessons about designing such monitoring systems (see Table 2). It is clear that we have a long way to go before we achieve a national monitoring system. Nevertheless, NICCQ has already yielded vital information. Current efforts are aimed at assessing differences within the metropolitan regions and addressing whether the differences across regions are attributable to demographics (that is, disparities in the care received by various sociodemographic groups) or to regional differences in the norms of care.