Development and Validation of a Grading System for the Quality of Cost-Effectiveness Studies

Published in: Medical Care, v. 41, no. 1, Jan. 2003, p. 32-44

Posted on on January 01, 2003

by Chiun-Fang Chiou, Joel W. Hay, Joel F. Wallace, Bernard S. Bloom, Peter J. Neumann, Sean D. Sullivan, Hsing-Ting Yu, Emmett B. Keeler, James M. Henning, Joshua J. Ofman

Read More

Access further information on this document at Medical Care

This article was published outside of RAND. The full text of the article can be found at the link above.

PURPOSE: To provide a practical quantitative tool for appraising the quality of cost-effectiveness (CE) studies. METHODS: A committee comprised of health economists selected a set of criteria for the instrument from an item pool. Data collected with a conjoint analysis survey on 120 international health economists were used to estimate weights for each criterion with a random effects regression model. To validate the grading system, a survey was sent to 60 individuals with health economics expertise. Participants first rated the quality of three CE studies on a visual analogue scale, and then evaluated each study using the grading system. Spearman rho and Wilcoxon tests were used to detect convergent validity and analysis of covariance (ANCOVA) for discriminant validity. Agreement between the global rating by experts and the grading system was also examined. RESULTS: Sixteen criteria were selected. Their coefficient estimates ranged from 1.2 to 8.9, with a sum of 93.5 on a 100-point scale. The only insignificant criterion was use of subgroup analyses. Both convergent validity and discriminant validity of the grading system were shown by the results of the Spearman rho (correlation coefficient = 0.78, P<0.0001), Wilcoxon test (P=0.53), and ANCOVA (F3,146 = 5.97, P = 0.001). The grading system had good agreement with global rating by experts. CONCLUSIONS: The instrument appears to be simple, internally consistent, and valid for measuring the perceived quality of CE studies. Applicability for use in clinical and resource allocation decisionmaking deserves further study.

This report is part of the RAND Corporation External publication series. Many RAND studies are published in peer-reviewed scholarly journals, as chapters in commercial books, or as documents published by other organizations.

Our mission to help improve policy and decisionmaking through research and analysis is enabled through our core values of quality and objectivity and our unwavering commitment to the highest level of integrity and ethical behavior. To help ensure our research and analysis are rigorous, objective, and nonpartisan, we subject our research publications to a robust and exacting quality-assurance process; avoid both the appearance and reality of financial and other conflicts of interest through staff training, project screening, and a policy of mandatory disclosure; and pursue transparency in our research engagements through our commitment to the open publication of our research findings and recommendations, disclosure of the source of funding of published research, and policies to ensure intellectual independence. For more information, visit

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.