Research Brief

Key Findings

  • A new prototype system for measuring and reporting the quality of cancer care in the United States has been developed and pilot-tested.
  • According to the NICCQ, patients with breast cancer in five U.S. major metropolitan areas receive about 86 percent of recommended care, and patients with colorectal cancer receive about 78 percent of recommended care.
  • Nevertheless, performance on individual measures of care varied widely.
  • Significant variation was observed across five major metropolitan areas for some of the care processes measured.
  • Despite its limitations, the quality monitoring system shows promise for revealing areas in which improvements in care are needed.

Breast-cancer patients in the United States have a 1-in-7 chance of missing some of the care they should be receiving, and colorectal cancer patients have a 1-in-5 chance of missing recommended care, according to a newly released RAND Health study that developed a prototype system for monitoring the quality of cancer care received by cancer patients. The good news is that they stand a good chance of receiving high-quality care, far better than the odds for the more long-term—and silent—chronic conditions, such as hypertension and diabetes, according to another recent RAND Health study. However, these odds represent averages: The chance of receiving some types of desirable care can be as low as 36 percent.

Why the Need for a Quality Monitoring System?

The Institute of Medicine (IOM), the medical research arm of the National Academy of Sciences, issued a report in 1999 that concluded that many cancer patients in the United States were not receiving state-of-the-art care. This finding was surprising to many oncologists because of the tremendous improvement in treatment knowledge that has resulted from clinical trials, the gold standard of research studies. However, research on quality has demonstrated that knowledge does not automatically translate to optimal care. The IOM raised concerns about the underuse and overuse of care and the fact that outcomes of treatment for the general population are not directly comparable to outcomes of clinical trials. It recommended a number of steps, including development of a quality monitoring system that would take advantage of recent improvements in our ability to measure quality.

In response to this challenge, researchers at RAND Health, in collaboration with the Harvard Medical School, relevant professional organizations, and a task force of expert advisers, have developed a cancer-care quality monitoring system. Called the National Initiative for Cancer Care Quality (NICCQ), this prototypical system measures and monitors care quality for two common cancers—breast cancer and colorectal cancer—and has been tested in five metropolitan areas.

Developing the Quality Monitoring System

The development of the NICCQ was guided by two requirements: the need to measure and report on the quality of cancer care as accurately as possible for the lowest achievable cost, and results that support and inform quality improvement efforts.

Based on discussions with clinical experts, professional societies, and other interested persons, the researchers deemed four key features to be critical in developing the NICCQ: a representative patient sample, assurance of privacy, valid measures of care quality, and multiple data sources.

Selecting Patients

For the pilot test of the monitoring system, a carefully designed selection protocol was needed to ensure that the patients whose care would be examined actually represent the population of patients with the same cancer diagnosis. Unlike clinical trials, which select participants who meet narrowly defined inclusion criteria (such as the absence of other illnesses, also known as comorbid conditions; membership in a particular age and/or ethnic group; or likelihood of survival), an accurate quality measurement system must capture a cross section of all patients with the condition of interest.

Also important was the source of participants. Each potential source—hospital cancer registries, insurance rosters, large-scale nationwide health surveys—offered both advantages and disadvantages. The researchers settled on the American College of Surgeons' National Cancer Database, because it obtains information on some 70 percent of all cancers nationwide, a larger proportion than can be obtained from alternative sources. A final consideration was whether to select only newly diagnosed patients or patients in all stages of cancer. The NICCQ decided to include only newly diagnosed patients, because the clinical care processes that produce the greatest effect on outcome often take place shortly after diagnosis.

Protecting Privacy

Protecting privacy and confidentiality is critical for cancer patients. For example, inappropriate disclosure of their diagnosis could compromise their employability. The NICCQ researchers found that a properly designed and conducted quality monitoring system poses little risk to patients.

Table 1. NICCQ Domains of Care

Domain Examples of Quality Measures
Diagnostic evaluation IF a patient has stage I to III breast cancer and had a breast tumor removed, THEN the pathology report should state the status of the margins.
Surgery IF a patient with stage I to III breast cancer undergoes mastectomy, THEN prior to undergoing mastectomy, the patient should be informed about the option of breast reconstruction after mastectomy.
Adjuvant therapy IF a patient with a diagnosis of stage I to III breast cancer has breast-conserving surgery, THEN the patient should receive local radiation therapy.
Management of treatment toxicity IF a patient ever receives highly emetogenic chemotherapy, THEN the patient should receive potent antiemetic therapy.
Post-treatment surveillance IF the patient has resection of a stage II or III colorectal cancer, THEN the patient should be counseled about the need to have first-degree relatives undergo colorectal cancer screening.

Defining Valid Measures of Cancer-Care Quality

It was imperative that the NICCQ include rigorously developed, validated measures of the quality of cancer care that had been demonstrated to improve outcomes, such as patients' likelihood of surviving, remaining disease free, or otherwise having a good quality of life. Choosing such measures for cancer-care quality presented several challenges. For example, an issue that always arises when developing quality measures is whether to measure the performance of care (process measures) or the results (outcome measures). Although it is easy to assume that the only true measure of quality is the outcome—improved survival or quality-adjusted life years—individual outcomes are very much affected by factors other than care quality, such as comorbid conditions. Thus, the researchers chose a combination of process and outcome measures, based on scientific evidence or, when such evidence was lacking, on expert consensus.

Another challenge, given rapidly evolving knowledge, is the need to ensure that measures are up to date—that is, that they reflect the latest treatment guidelines. Yet, at the same time, guidelines used to assess care provided in the past must reflect the standards of that time.

The project team developed measures of care processes within a set of five domains that represent important aspects of cancer care. (Table 1 shows these domains, along with an example of a measure from each category.) The measures specify events or situations that should cause a clinician to recommend a particular type of clinical care—a diagnostic test, treatment, or other intervention—for a particular patient, given the available evidence or expert consensus. In addition, adherence was measured for eight components of care that cut across all five domains: (1) testing; (2) pathology; (3) documentation; (4) referral; (5) timing; (6) receipt of treatment; (7) technical quality; and (8) respect for patient preferences.

Collecting Data from Multiple Sources

The information needed for a comprehensive assessment of a cancer patient's care often must be obtained from a number of different sources, such as the patient's hospital records, the records of several different individual physicians or outpatient clinics, tumor registries, and even the patients themselves, because no single source has all the information relevant to a patient's treatment. Thus, a third requirement for the NICCQ was a comprehensive and ongoing data-collection protocol that relies on several sources of available data: registries, medical records, and patient surveys. Patient surveys are particularly important for capturing information about current health status, care experiences, involvement in decisionmaking, quality of life, functional outcomes, and coordination of care—information that is rarely available from medical records.

The NICCQ collected data in two ways: via a patient survey and via review of medical records. Administration of the patient survey first addressed a number of logistical challenges, such as gaining patient consent and permission for record release. Further, in addition to providing information about their experiences and outcomes of care, patients were asked to provide the names and addresses of doctors, hospitals, and other providers involved in their care, which proved to be the most effective way to identify relevant medical records. Researchers accessed the medical records of the doctors named by the patients, as well as records of any other doctors named in those initial records. Data obtained from medical records included detailed information about tumor characteristics, staging, referrals and decisionmaking, initial treatment as well as any additional treatments, and comorbid conditions from 3 months before diagnosis through 12 months after the diagnosis.

For a patient to be eligible for a particular measure, the patient's data file had to include the records for the specialist generally responsible for providing the care being measured and the type of document that would normally show the type of information in question. For example, the measure "IF a patient is treated with chemotherapy, THEN body-surface area should be documented" would apply only to a patient whose medical record included the records of the medical oncologist and the chemotherapy flowsheet.

Quality Is Good but Highly Variable

The Patients and the Measures

Using hospital registries, the researchers recruited 400 patients with either breast or colorectal cancer from each of five metropolitan areas—Atlanta, Cleveland, Houston, Kansas City, and Los Angeles. Participation was limited to those who were between 21 and 80 years of age at the time of diagnosis, were alive at the time of contact (approximately four years after diagnosis), and were able to speak and write English.

For breast cancer, the researchers defined 36 quality measures. On average, breast-cancer patients received 86 percent of recommended care. Adherence to recommended care within the five clinical care domains varied somewhat, ranging from 73 percent (management of treatment toxicity) to 94 percent (post-treatment surveillance). However, adherence for different components of care ranged much more widely, from 13 percent (referral) to 60 percent (documentation) to 96 percent (testing) and 97 percent (receipt of treatment).

For colorectal cancer, the researchers defined 25 quality measures. With respect to these measures, patients received an average of 78 percent of recommended care. Adherence to recommended care within the five care domains ranged from 50 percent (post-treatment surveillance) to 93 percent (surgery). Adherence across the different components of care varied somewhat less, ranging from 57 percent (respect for patient preferences) and 59 percent (referral) to 93 percent (pathology).

Variation in Quality Across Metropolitan Areas

To find out whether quality varied by region, the researchers compared adherence to recommended care among the five metropolitan areas.

The average overall adherence for each of the metropolitan areas ranged from 82 to 87 percent, and no one region consistently provided better care across all domains. When specific types of care were compared among the metropolitan areas, the greatest variation was seen for breast-cancer surgery, followed by diagnostic evaluation and adjuvant therapy for breast cancer.

Among the 36 quality measures for breast-cancer care, adherence was less than 85 percent for half: Regional variation was significant for seven of those measures. For example, among patients treated with chemotherapy, the planned dose was consistent with published guidelines for only 61 percent of the patients in the study, a figure that ranged from 32 to 75 percent across metropolitan areas. Interestingly, six of the 18 measures with less than 85 percent adherence pertained to chart documentation of key clinical factors that are believed to be essential for providing appropriate cancer care.

Among the 25 quality measures for colorectal cancer, adherence was less than 85 percent for 14, but only one showed significant regional variation (radiation therapy for stage II or III rectal cancer). Only 66 percent of all stage II and III patients received recommended radiation therapy, and this figure varied from 58 to 92 percent across metropolitan areas.

Table 2. Lessons Learned from the NICCQ

  • Developing quality measures has not been limited by lack of scientific evidence, but not all aspects of cancer care have been evaluated in trials. Thus, whereas some measures are based on clinical evidence, others are based on consensus statements and expert opinion, suggesting the need for more clinical trials.
  • As sources of patients for a national quality monitoring system, all currently active cancer registries have shortcomings. Time delays—the time required for a case to be entered into the registry—are a key barrier: Registry strategies that include rapid case ascertainment (an accelerated process for determining that a patient has cancer), collection of a few key clinical data elements, and use of structured standardized electronic data collection and reporting could address this shortcoming.
  • Financial turmoil affecting health care institutions may have an adverse effect on a national monitoring system. Hospital registries may find it difficult to provide data on care quality while fulfilling their core obligations to care for patients.
  • The need to obtain patient consent implies that even the best selection strategy cannot provide perfectly representative samples of patients. Participation and response rates could be enhanced by earlier case ascertainment and contact with patients. Also helpful would be streamlining the Institutional Review Board (IRB) process—that is, the mandatory review each institution must conduct for surveys used to collect patient information so that privacy, confidentiality, and avoidance of harm are ensured. In any case, statistical adjustment will be needed to correct for patient and hospital nonparticipation and nonresponse. Local IRBs should be encouraged to delegate review to a central organization.
  • One issue that must be resolved is how long to collect data. An important contribution of the NICCQ was that data were collected up to four years after diagnosis. Limiting data collection would make it impossible to obtain information on care processes that continue for many months or even years, such as adjuvant chemotherapy, as well as on long-term outcomes.
  • Quality monitoring cannot have much effect on patient outcomes unless efforts to improve quality ensue. Previous regional quality improvement efforts could serve as a useful framework.
  • Implementation of a national quality monitoring system is not a trivial undertaking. It will require, among other things, substantial resources and sustained commitment. Although it may not be feasible to implement such a system in the near future, expanding the existing prototype in selected states may be realistic over time.

What Do the Findings Mean for the Quality of Cancer Care?

The current quality of care for breast and colorectal cancers clearly contains some good news and some bad news. The good news is that a large proportion of patients is receiving recommended care. Adherence actually approached 100 percent for several measures, showing that excellent quality of care is possible. However, the bad news is that adherence was less than 85 percent for nearly half the measures, and for some measures, the quality varied substantially from one metropolitan area to another.

Cancer-care quality appears better than care for other chronic illnesses, according to several of RAND Health's recent studies, which suggest that people receive recommended care, on average, only about half the time. The findings also suggest that the quality of cancer care is better than earlier studies would indicate. There may be several possible reasons.

First, in contrast to prior studies, the NICCQ study used explicit quality measures with strictly defined eligibility criteria. Simply measuring utilization rates probably underestimates the appropriate use of care, because it fails to take into account how many patients are actually eligible for the process in question.

Second, few other studies have used the comprehensive data-collection methods used in the NICCQ. If many of the records used for the study were actually missing data (such as documentation of a treatment provided), care might appear to be of poorer quality than it actually was, since treatments might have been administered but not documented in the records used for data collection, a particularly likely possibility for care delivered in outpatient settings.

A third, related problem is that the evidence necessary for determining eligibility for a particular element of care might not have been in the records. However, the NICCQ went to great lengths to ensure complete records, which included contacting over 4,000 physicians.

If cancer-care quality is indeed higher than that for other illnesses, why? Perhaps the urgency of a new cancer diagnosis focuses providers' and patients' attention on treatment in a way that chronic diseases (especially "silent" ones, such as hypertension and diabetes) do not. Another possibility is that providers of cancer care might have greater confidence in the effectiveness of the care they provide than do physicians in other specialties. Still, the findings do demonstrate many areas in which care quality could be improved, such as the use of appropriate doses, appropriate timing of treatment initiation, and accurate documentation of the information needed for treatment and care.

The evaluation of the NICCQ was not without its limitations. First, the assessment was limited to two types of cancer and to certain stages of those cancers. Care quality might be different for other types of cancer or for more- or less-advanced breast and colorectal cancer. Second, the selected measures reflect both scientific evidence and expert consensus. Differences in the strength of the evidence as perceived by physicians could affect performance of particular measures. Third, the study excluded patients who could not read English or who had died shortly after diagnosis, two groups who might have received lower-quality care. Fourth, nonparticipation of particular hospitals or patients could also limit the generalizability of the findings.

Lessons Learned

In spite of its limitations, the NICCQ provided a snapshot of the quality of cancer care across the United States, as well as some valuable lessons about designing such monitoring systems (see Table 2). It is clear that we have a long way to go before we achieve a national monitoring system. Nevertheless, NICCQ has already yielded vital information. Current efforts are aimed at assessing differences within the metropolitan regions and addressing whether the differences across regions are attributable to demographics (that is, disparities in the care received by various sociodemographic groups) or to regional differences in the norms of care.

This report is part of the RAND Corporation research brief series. RAND research briefs present policy-oriented summaries of individual published, peer-reviewed documents or of a body of published work.

Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.