Deciding Not to Measure Performance

The Case of Acute Otitis Media

Published in: Joint Commission Journal on Quality and Safety, v. 29, no. 1, Jan. 2003, p. 27-36

by Rita Mangione-Smith, Karen Onstad, Lok Wong, Joachim Roski

BACKGROUND: No Health Plan Employer Data and Information Set (HEDIS) performance measures evaluate health plans for possible overuse or inappropriate use of antibiotics. Acute otitis media (AOM), or infection of the middle ear, is one of the most common infections in children. The antibiotic resistance of the bacteria that cause AOM and the general overuse and inappropriate use of antibiotics for this condition are taking center stage as a major public health threat. An effort was undertaken to develop a new HEDIS performance measure that evaluates appropriate antibiotic use in children with AOM. THE MEASURE DEVELOPMENT PROCESS: The measure development process has three major phases: the initial development phase, the field-testing and analysis phase, and the measure refinement phase. With AOM, the measure development process could not proceed beyond the first phase for reasons that are be discussed in detail. CONCLUSIONS: Additional difficulties beyond feasibility issues may arise in developing a quality of care performance measure. The measure development process discussed in this article failed primarily because of issues related to relevance as well as the scientific soundness of the proposed measures. If the evidence base related to the diagnosis and management of a particular condition is sparse, no standard can be developed against which to measure performance, and the criterion of scientific soundness cannot be met. This is the case with AOM. Unfortunately, the evidence base will likely remain inadequate to support the development of such a measure in the future.

This report is part of the RAND Corporation External publication series. Many RAND studies are published in peer-reviewed scholarly journals, as chapters in commercial books, or as documents published by other organizations.

Our mission to help improve policy and decisionmaking through research and analysis is enabled through our core values of quality and objectivity and our unwavering commitment to the highest level of integrity and ethical behavior. To help ensure our research and analysis are rigorous, objective, and nonpartisan, we subject our research publications to a robust and exacting quality-assurance process; avoid both the appearance and reality of financial and other conflicts of interest through staff training, project screening, and a policy of mandatory disclosure; and pursue transparency in our research engagements through our commitment to the open publication of our research findings and recommendations, disclosure of the source of funding of published research, and policies to ensure intellectual independence. For more information, visit

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.