Cover: ExpertLens


A System for Eliciting Opinions from a Large Pool of Non-Collocated Experts with Diverse Knowledge

Published in: Technological Forecasting and Social Change, v. 78, no. 8, Oct. 2011, p. 1426-1444

Posted on 2011

by Siddhartha Dalal, Dmitry Khodyakov, Ramesh Srinivasan, Susan G. Straus, John L. Adams

The complexity of policy decision-making raises the need to elicit opinions from large and heterogeneous groups of stakeholders with broad and diverse sets of expertise. Existing options for elicitation include small face-to-face panels of experts by using the Nominal Group Technique (NGT), large Delphi panels whose members do not interact with each other face-to-face, and crowdsourcing, which involves an open call for input issued to a large community of people. In an attempt to close the gap between the practical needs of policy makers and the methodological challenges associated with eliciting opinions of large, diverse, and distributed groups, we have developed a new online elicitation system and methodology called ExpertLens. By optimizing the direct interactions of NGT with the larger number of Delphi participants and the wisdom of "selected crowds," our approach is designed to save on the costs associated with traditional expert panels, while increasing accuracy in elicitation by reducing the potential for group process losses that can occur in large, diverse, and non-collocated panels whose members interact via asynchronous online discussion boards. The ExpertLens approach is iterative, does not require participants to develop consensus, and determines what the group "thinks" by statistically analyzing data collected in all rounds of the elicitation. This paper describes the ExpertLens system and methodology, briefly discusses recent ExpertLens trials, provides conceptual arguments for why it is an appropriate model for eliciting expert opinions, illustrates its main components and analytics by using an infrastructure investment example, and discusses a research agenda for testing the underlying tenets of the ExpertLens approach.

This report is part of the RAND external publication series. Many RAND studies are published in peer-reviewed scholarly journals, as chapters in commercial books, or as documents published by other organizations.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.