Cover: Lack of Preregistered Analysis Plans Allows Unacceptable Data Mining for and Selective Reporting of Consensus in Delphi Studies

Lack of Preregistered Analysis Plans Allows Unacceptable Data Mining for and Selective Reporting of Consensus in Delphi Studies

Published in: Journal of Clinical Epidemiology, Volume 99 (July 2018), Pages 96-105. doi:10.1016/j.jclinepi.2018.03.007

Posted on rand.org Apr 17, 2018

by Sean Grant, Marika Booth, Dmitry Khodyakov

Objectives

To empirically demonstrate how undisclosed analytic flexibility provides substantial latitude for data mining and selective reporting of consensus in Delphi processes.

Study Design and Setting

Pooling data across eight online modified-Delphi panels, we first calculated the percentage of items reaching consensus according to descriptive analysis procedures commonly used in health research but selected post hoc in this article. We then examined the variability of items reaching consensus across panels.

Results

Pooling all panel data, the percentage of items reaching consensus ranged from 0% to 84%, depending on the analysis procedure. Comparing data across panels, variability in the percentage of items reaching consensus for each analysis procedure ranged from 0 (i.e., all panels had the same percentage of items reaching consensus for a given analysis procedure) to 83 (i.e., panels had a range of 11% to 94% of items reaching consensus for a given analysis procedure). Of 200 total panel-by-analysis-procedure configurations, four configurations (2%) had all items and 64 (32%) had no items reaching consensus.

Conclusion

Undisclosed analytic flexibility makes it unacceptably easy to data mine for and selectively report consensus in Delphi processes. As a solution, we recommend prospective, complete registration of preanalysis plans for consensus-oriented Delphi processes in health research.

This report is part of the RAND external publication series. Many RAND studies are published in peer-reviewed scholarly journals, as chapters in commercial books, or as documents published by other organizations.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.