Developing a Natural Language Processing Application for Measuring the Quality of Colonoscopy Procedures

Published in: Journal of the American Medical Informatics Association, v. 18, suppl 1, Dec. 2011, p. 150-156

Posted on RAND.org on December 01, 2011

by Henk Harkema, Wendy W. Chapman, Melissa Saul, Evan S. Dellon, Robert E. Schoen, Ateev Mehrotra

Read More

Access further information on this document at Journal of the American Medical Informatics Association

This article was published outside of RAND. The full text of the article can be found at the link above.

OBJECTIVE: The quality of colonoscopy procedures for colorectal cancer screening is often inadequate and varies widely among physicians. Routine measurement of quality is limited by the costs of manual review of free-text patient charts. Our goal was to develop a natural language processing (NLP) application to measure colonoscopy quality. MATERIALS AND METHODS: Using a set of quality measures published by physician specialty societies, we implemented an NLP engine that extracts 21 variables for 19 quality measures from free-text colonoscopy and pathology reports. We evaluated the performance of the NLP engine on a test set of 453 colonoscopy reports and 226 pathology reports, considering accuracy in extracting the values of the target variables from text, and the reliability of the outcomes of the quality measures as computed from the NLP-extracted information. RESULTS: The average accuracy of the NLP engine over all variables was 0.89 (range: 0.62e1.0) and the average F measure over all variables was 0.74 (range: 0.49e0.89). The average agreement score, measured as Cohen's k, between the manually established and NLP-derived outcomes of the quality measures was 0.62 (range: 0.09e0.86). DISCUSSION: For nine of the 19 colonoscopy quality measures, the agreement score was 0.70 or above, which we consider a sufficient score for the NLP-derived outcomes of these measures to be practically useful for quality measurement. CONCLUSION: The use of NLP for information extraction from free-text colonoscopy and pathology reports creates opportunities for large scale, routine quality measurement, which can support quality improvement in colonoscopy care.

This report is part of the RAND Corporation external publication series. Many RAND studies are published in peer-reviewed scholarly journals, as chapters in commercial books, or as documents published by other organizations.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.