Read Online Version

Download Free Electronic Document

FormatFile SizeNotes
PDF file 0.1 MB

Use Adobe Acrobat Reader version 10 or higher for the best experience.

Research Brief

Key findings:

  • A team from RAND and the University HealthSystem Consortium (UHC) developed a toolkit to help hospitals enhance their quality improvement efforts using quality indicators from the Agency for Healthcare Research and Quality.
  • The RAND-UHC evaluation found that most of the participating hospitals judged the tools to be useful in supporting quality improvement. They also provided feedback on the tools, which was used to revise the toolkit to enhance its usefulness and usability.
  • Improved performance depended on leadership buy-in: Hospitals with board and executive support were able to do more.
  • The toolkit also helped achieve staff consensus on the extent of quality gaps and supported the use of evidence-based practices.

Quality of care in U.S. hospitals is a significant problem. The pace of quality improvement efforts in hospitals has picked up in recent years, with a focus on monitoring and public reporting of quality information. Prominent among such tools are quality indicators (QIs) developed by the Agency for Healthcare Research and Quality (AHRQ), including two sets of indicators for inpatient settings: the Inpatient Quality Indicators (IQIs) and Patient Safety Indicators (PSIs). However, if hospitals lack the needed knowledge, skills, or resources, their efforts to improve performance may have little effect on quality outcomes. Many hospitals may need help to enhance their capacity to use QIs for effective quality improvement.

To meet this need, AHRQ contracted with a team of researchers from RAND and the University HealthSystem Consortium (UHC), a large collection of academic medical centers, to develop a toolkit for hospitals to support their quality improvement efforts based on the AHRQ QIs. This toolkit provides a roadmap to guide quality improvement. It includes a sequence of steps designed to help hospitals calculate desired rates for the QIs, set priorities for improvement, develop specific strategies and goals, implement these strategies, and sustain improvements. This research highlight describes this toolkit, reports findings from RAND-UHC's field-testing of the toolkit in hospital settings, and highlights ways hospitals can use the toolkit to improve their quality of care.

Developing the Toolkit

To develop the toolkit, the RAND-UHC team first reviewed an extensive literature on hospital quality improvement. First, the review assessed the characteristics and performance of AHRQ's QIs. Second, it documented the many ways hospitals are already using the QIs. Third, the review identified key factors that influence success in implementing improvements.

The RAND-UHC team convened an expert panel to help identify and develop tools. Based on the literature review and the panel guidance, the RAND-UHC team produced an alpha version of the toolkit (see table).

Field-Testing the Toolkit

To assess the usability of the candidate tools, the RAND-UHC team field-tested the kit to evaluate its helpfulness for hospitals for implementing performance improvement interventions based on the AHRQ QIs. Six hospitals participated in a formal evaluation; four others contributed informal feedback. Hospitals represented a mix of characteristics, including bed size, academic/community status, geographic diversity (regional and urban/rural), and safety net status. RAND collected primary data from the participating hospitals on improvement experiences and toolkit usability. Data were collected using pre- and post-test interviews, follow-up interviews, and usability testing interviews.

Most hospitals reported that the tools helped support improvement processes. They varied in the tools they used, and some reported using none of the tools because they already had well-established quality improvement processes and tools. Here are other key findings from field-testing:

  • All of the hospitals focused on patient safety (the PSIs) rather than more traditional approaches to quality improvement (the IQIs). These decisions appeared to be responses to performance expectations from external parties, such as Medicare, other payers, and regulatory entities.
  • For many hospitals, their first challenge was uncertainty about the accuracy of rates calculated for their PSIs, due to issues in documentation and coding of health records. Confidence in the accuracy of PSI rates was a prerequisite because a loss of credibility with clinical staff can irreparably damage a quality improvement project.
  • Improved performance depended on leadership buy-in: Hospitals with board and executive support could do more. Hospitals cited external incentive programs based on the AHRQ QIs (such as public reporting or pay for performance) as powerful motivators for hospital boards.
  • Several hospitals found that the toolkit helped achieve staff consensus on the extent of quality gaps and supported the use of evidence-based practices.

Hospitals provided feedback on existing tools and also requested additions to the toolkit. They asked for (1) a tool that provides guidance on documenting and coding medical record data for use in calculating PSIs; (2) examples of how to set priorities; and (3) guidance on best practices for additional PSIs (beyond what the alpha toolkit contained). Using this feedback, the RAND-UHC team revised existing tools and added new tools to the toolkit (see table).

A Toolkit for Hospital Quality Improvement
Improvement Steps for Hospitals Key Tools
Determining readiness for change
  • Fact sheets on IQIs and PSIs
  • Presentations on QIs
  • "Getting Ready for Change" self-assessment
Applying QIs to hospital data
  • Applying AHRQ QIs to hospital data
  • Documentation and coding for PSIs
  • Assessing indicator rates using trends and benchmarksa
Identifying priorities for quality improvement
  • Prioritization matrix
  • Prioritization matrix examplea
Implementing improvement
  • Improvement methods overview
  • Project charter
  • Examples of PSI improvement strategies
  • Selected best practices and suggestions for improvementsb
  • Gap analysis
  • Implementation plan and measurement
  • Project evaluation and debriefing
Monitoring progress for sustainable improvement
  • Guidance on monitoring progress
Analyzing return on investment
  • Estimating return on investment
Using other resources
  • Available quality improvement guides
  • Specific tools to support change
  • Case study of PSI improvement implementationa
a Added in response to hospital feedback.
b The alpha toolkit addressed three PSIs; five more were added with revisions.
SOURCE: "Introduction and Roadmap" and "Webinar" in Farley D, Weinick R, Mayer L, Cerese J, Burns R, and Hussey P, AHRQ Quality Indicators™ Toolkit for Hospitals, Rockville, Md.: Agency for Healthcare Research and Quality, March 2012 (http://www.ahrq.gov/qual/qitoolkit/index.html).

Concluding Observations

This project confirmed that hospitals face challenges in quality improvement and found that the toolkit was helpful in addressing them. Hospitals judged many of the tools to be user-friendly, and the RAND-UHC team employed their feedback to enhance the tools' utility. In using the QIs, however, hospitals should be aware of known validity issues and not overinterpret results. As their name suggests, the QIs are "indicators" that point to areas of potential strengths or problems. Hospitals should supplement them with additional examination of care processes and data analysis before deciding on improvement actions. square

This report is part of the RAND Corporation research brief series. RAND research briefs present policy-oriented summaries of individual published, peer-reviewed documents or of a body of published work.

Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.