Research Brief

Changes in health care organization and financing in the United States have increased consumers' concerns about the quality of the health care they receive. Among national efforts to improve the quality of health care, perhaps none has been as prominent as the movement to evaluate health care quality and to disclose the results of that evaluation publicly, usually in the form of "report cards," "provider profiles," or "consumer reports," sometimes communicated via the World Wide Web. The purpose of public disclosure of information on quality is twofold: to facilitate informed choice and to stimulate quality improvement. Researchers at RAND, who have been among those involved in designing health plan performance evaluation and reporting systems for several of the nation's largest employers, have begun evaluating the effects of those systems. While it was generally assumed by health policy planners that consumers would welcome access to these report cards, the existing research seems to suggest that this is not the case. In order to learn why those faced with health care decisions fail to use comparative performance information and what can be done to increase its use, we have examined

  • how information for report cards is gathered and organized;
  • who uses them and who does not;
  • why people don't use report cards; and
  • what might be done to make them more user friendly.

Health care report cards are constructed from information gathered from several national surveys of health care quality. Contrary to expectations, these report cards appear to affect the health care decisionmaking of large health care providers more than that of individual consumers. We have learned through our own experience and that of others that the way report cards organize and present their evaluations strongly influences whether the report cards are used. Moreover, a few simple techniques of effective communication can improve their usability.

How Are Health Care Report Cards Created?

Over the past decade, collection and release of comparative data on hospitals, health plans, and individual physicians have been increasing, a reflection of the increasing demand for disclosure and accountability in all walks of life. The information provided includes reputation, mortality rates (for specific procedures), accreditation status, and results of some annual surveys by organizations that evaluate health care quality.

Providing such information to the public requires the systematic collection and evaluation of data on quality of care. These evaluations include comparisons of performance over time, among providers, and against established standards.

The construction of all health care report cards involves a similar set of steps. The first step is to build a database of standardized performance measures. Two of the key sets of measures about health plan performance, for example, are the Health Plan Employer Data and Information Set (HEDIS) and the Consumer Assessments of Health Plans Survey (CAHPS). HEDIS was designed and implemented by the National Committee for Quality Assurance (NCQA), a nonprofit organization that is the nation's leading accreditor of health plans; CAHPS was designed by leading researchers at RAND, the Research Triangle Institute, and Harvard University with support from the Agency for Healthcare Research and Quality. Health plans supply HEDIS data to NCQA, whereas third-party vendors collect CAHPS data from health plan enrollees. Table 1 shows the kinds of information these data sets capture: evaluations of the quality of care processes (based on indicators of effectiveness in preventing and treating disease), access to care, treatment outcome, and patient satisfaction (using the results of CAHPS).

Table 1. Kinds of Public Information Released by Providers

Provider Type Type of Information Examples
Hospital Overall mortality HCFA
Mortality by specific procedure New York; Pennsylvania; Pacific Business Group on Health (PBGH)
Mortality by specific disease California Hospital Outcomes Project; Cleveland Quality Choice
Accreditation Joint Commission on the Accreditation of Healthcare Organizations (JCAHO)
Rankings by specialty area U.S. News & World Report
Health plan Process of care NCQA (HEDIS)
Patient experience with care NCQA (CAHPS)
Accreditation NCQA
Categorical rankings CARS, U.S. News &World Report
Medical group Process of care PBGH
Patient experience with care PBGH
Physician Mortality rate for coronary artery bypass graft surgery New York; Pennsylvania

Source: Adapted from McGlynn and Adams, 2001.

The second step in constructing a report card is to organize the information to be communicated into a hierarchy of categories called a framework. People tend to remember and use information more easily when it is presented in a small number of bits, so organizing the vast array of evaluation data into a small number of categories increases the usability of report cards.

The final steps in the process consist of analyzing and summarizing the data. In order to make meaningful comparisons, researchers must devise methods to estimate the values for missing data, adjust outcome data for the prior health status of the patients included (a technique known as risk adjustment), and evaluate various aspects of care on similar scales.

The CARS and HCFA Experiences

In 1998, we assisted the "Big Three" U.S. automobile manufacturers and the United Auto Workers (UAW) in creating a Coordinated Autos/UAW Reporting System (CARS) for evaluating the performance of health plans that contracted with one or more auto manufacturers. Using a similar process, we demonstrated how a set of performance reports designed for the Health Care Financing Administration (HCFA) could help Medicare beneficiaries choose among health plans. When we reviewed the methods used by others to summarize data for reporting systems, we discovered wide variations in the methods used to summarize performance data. Although the methods were all valid, the variations led to significantly different conclusions from the same data. This finding raises the concern that if multiple report cards are issued in the same geographic area, their evaluations may conflict, confusing the public and leading to mistrust of the entire process. Developing a national consensus on reporting strategies would decrease the likelihood that consumers would be presented with conflicting information about the same plans or providers and would increase the likelihood that people would trust and use the information.

What Have We Learned About How People and Organizations Use Report Cards?

Given the now widespread existence of health care report cards, what is known about who uses them, how they are used, and their effectiveness? In collaboration with researchers in the United Kingdom, we reviewed all published evaluations of U.S. reporting systems. We discovered that evaluation of report cards has not kept pace with their development. Although there is now a major industry centered around the collection, analysis, and publication of quality data, attempts to evaluate the effectiveness of these efforts have been sparse and difficult to conduct. Further, because no clear agreement exists about their actual purpose, evaluation of their effectiveness is difficult.

Report Cards Seem to Have Little Influence Among Consumers and Physicians

Although evidence from surveys and focus groups suggests that consumers want more information about the performance of their health care providers, few individuals are apparently influenced by the information in their decisionmaking. According to the surveys, consumers' choice of hospitals relied more on anecdotal press reports of adverse events than on the comparative assessments that were available. Only one study apparently showed any significant effect of public disclosure on consumer decisionmaking. This study evaluated the effects of a New York state reporting system for physician and hospital performance on coronary artery bypass graft (CABG) surgery mortality and choice of physicians and hospitals in the early 1990s. CABG is one of the primary cardiac revascularization procedures used to treat coronary artery disease. During the study period, actual and risk-adjusted mortality rates decreased significantly. Those physicians and hospitals that performed the surgery more frequently and with lower mortality experienced an increase in their market share compared with lower-volume, higher-mortality providers.

Large-scale purchasers of corporate health plans are no more likely than individual consumers to use report cards in their decisionmaking. Only a small proportion of purchasers reported even using accreditation data in their purchasing decisions.

Physicians also are unlikely to use publicly available information as the basis for patient referrals. Although physicians tend to consider the information accurate, they seldom share it with patients. Moreover, many physicians believe that public disclosure of performance information encourages other physicians to refuse to treat those patients in the poorest health. A review of the results of the CABG study by independent researchers found that the better-performing providers were not turning away higher-risk patients. Nevertheless, report cards that evaluate physicians, hospitals, or other provider groups based on treatment outcomes need to employ appropriate risk adjustment methods to ensure that the prior health status of individual patients has been taken into account.

Health Provider Organizations Have Responded Favorably

In contrast to consumers and physicians, health provider organizations such as hospitals seem to respond to comparative performance data like that provided by the CABG study. Studies show that in response to such feedback, hospitals have demonstrated improvement in performance of a variety of procedures and in patient outcomes. For example, the 2001 NCQA report, The State of Managed Care Quality, reported improvements from 1999 to 2000 in a number of quality indicators for managed care organizations. These indicators included comprehensive diabetes care (such as eye exams and blood sugar monitoring), women's health measures (such as screening for breast and cervical cancer), childhood immunizations, smoking cessation advice, and several measures of heart disease care.

These findings suggest that future work should focus at least in part on optimizing the design and dissemination of report cards for health provider organizations. However, it is not yet clear whether information that is disclosed to the public has a greater impact on organizational performance than information that is shared only among members of provider organizations. The assumption has been that hospitals improve their procedures and outcomes in response to consumer demand, or out of fear that consumers will react to negative publicity by taking their business elsewhere. If that is the case, the effect of health care report cards on the one group that has proven most responsive to them might begin to wane when it discovers that few consumers, individual or corporate, are paying attention to the negative publicity that the report cards generate. Furthermore, researchers must focus their attention on just why it is that consumers fail to use report cards and what can be done to improve their use.

What Keeps People from Using Report Cards?

One of the primary reasons for the public disclosure of comparative health care assessment is the belief that informed consumers would be empowered to make better decisions about their health care and that such informed choices will lead to better health care. In surveys and focus groups, consumers say they want more information on which to base decisions about health care. But why do they fail to use the information they receive? Consumer surveys have uncovered several reasons. Frequently, consumers report having difficulty understanding the information they are given because of its technical nature and the way it is presented. Consumers also report that the types of information available are not relevant to the decisions they need to make, or that too much information is presented and no guidance is provided on how to sort through it. Studies of how people process and use information to make decisions show that up to a point, people's confidence in their decisions increases with the amount of information they are given on which to base those decisions; however, past that point, providing more information actually causes people's confidence to wane.

Additional concerns are that the information is not available when it is needed, that too little time is available to review the material, and that information furnished by the health plans and providers themselves may not be trustworthy. The surveys show that individual consumers rely instead on information from family and friends when making health care decisions.

What Can Be Done to Increase the Usability of Report Cards?

Cognitive psychologists have learned a great deal about how people acquire and use new information and about the effectiveness of various communication styles. We reviewed a number of the report cards that were available to assess how they presented their information relative to what we know about effective communication styles. A key problem for many of these report cards appeared to be poor presentation of information. Because the use of health care report cards to help with decisionmaking is a voluntary activity, users are not compelled to continue working with material that does not sustain their interest.

We identified a variety of ways that researchers and authors could increase the usability of report cards:

Engage the reader at the outset by providing a familiar context for the information being presented. Material has often been presented with no cues about its organization or how to use it; the report cards sometimes lacked even a uniform organizational structure. Research shows that new innovations take time to diffuse into the mainstream. Thus, if employers, health plan purchasers, and others want to increase the usability of health care report cards now, effective communications techniques such as those described in Table 2 will need to be adopted.

Table 2. Techniques to Increase the Effectiveness of Health Care Report Cards

  • Create the framework of the reporting system by identify- ing the general categories of information that are most important to consumers (that is, use a top-down approach).
  • Provide a clear context up front for the information being presented and the reasons for its presentation, to enable consumers to assimilate the information into what they already know or believe.
  • Organize the information that is presented to the consumer with a clear, consistent structure, and provide cues to that structure with headings.
  • Choose a structure that allows consumers to access the information they want without having to read through the entire report.
  • Present information in more than one way; if graphics are used, choose those most appropriate to the kind of infor- mation being presented (that is, those that make differences easy to perceive).

Source: Adapted from McGlynn, Adams, Hicks, and Klein, 1999.

Provide specific educational initiatives for target audiences. Such initiatives may include use of the mass media to educate the public, use of continuing professional education to educate health care professionals, and release of the data as part of an educational package aimed at providers to promote quality improvement.

Conduct more comprehensive assessments of the impact of public disclosure of information. These assessments must identify who actually uses the information and for what purposes. Identifying these unknowns will allow the creators of health care report cards to tailor the messages they present more effectively and to reach a much larger segment of the potential users. Our findings thus far suggest that the optimal format for report cards targeted to health care providers and plan purchasers will differ considerably from the format that will appeal most to consumers of health care.

This report is part of the RAND Corporation research brief series. RAND research briefs present policy-oriented summaries of individual published, peer-reviewed documents or of a body of published work.

Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.