Measuring Success for Health Care Quality Improvement Interventions

BACKGROUND: The lack of a standard measure of quality improvement (QI) success and the use of subjective or self-reported measures of QI success has constrained efforts to formally evaluate QI programs and to understand how the various contextual factors impact QI success. OBJECTIVES: The objective of this study was to assess how best to measure "QI success" by comparing self-reported and externally rated measures of QI success. RESEARCH DESIGN: We performed a retrospective evaluation that analyzed data on different measures of QI success for organizations after their staff completed the QI training. SUBJECTS: The sample included 30 organizations whose staff had received QI training during 2006-2008, and who had used this training to carry out at least some subsequent QI initiative in their organizations. MEASURES: We developed 2 measures of self-reported QI success based on survey responses and 4 externally rated measures of QI success based on outcome data provided by the participating organizations in addition to qualitative data generated from the interviews. RESULTS: We found some variation in the mean scores of the different QI success measures and only moderate to small correlations between the self-report and externally rated QI measures. CONCLUSIONS: This study confirms that there are important differences between self-reported and externally rated measures of QI success and provides researchers with a methodology and criteria to externally rate measures of QI success.