Deciding Not to Measure Performance

The Case of Acute Otitis Media

Published in: Joint Commission Journal on Quality and Safety, v. 29, no. 1, Jan. 2003, p. 27-36

Posted on RAND.org on January 01, 2003

by Rita Mangione-Smith, Karen Onstad, Lok Wong, Joachim Roski

BACKGROUND: No Health Plan Employer Data and Information Set (HEDIS) performance measures evaluate health plans for possible overuse or inappropriate use of antibiotics. Acute otitis media (AOM), or infection of the middle ear, is one of the most common infections in children. The antibiotic resistance of the bacteria that cause AOM and the general overuse and inappropriate use of antibiotics for this condition are taking center stage as a major public health threat. An effort was undertaken to develop a new HEDIS performance measure that evaluates appropriate antibiotic use in children with AOM. THE MEASURE DEVELOPMENT PROCESS: The measure development process has three major phases: the initial development phase, the field-testing and analysis phase, and the measure refinement phase. With AOM, the measure development process could not proceed beyond the first phase for reasons that are be discussed in detail. CONCLUSIONS: Additional difficulties beyond feasibility issues may arise in developing a quality of care performance measure. The measure development process discussed in this article failed primarily because of issues related to relevance as well as the scientific soundness of the proposed measures. If the evidence base related to the diagnosis and management of a particular condition is sparse, no standard can be developed against which to measure performance, and the criterion of scientific soundness cannot be met. This is the case with AOM. Unfortunately, the evidence base will likely remain inadequate to support the development of such a measure in the future.

This report is part of the RAND Corporation external publication series. Many RAND studies are published in peer-reviewed scholarly journals, as chapters in commercial books, or as documents published by other organizations.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.