How ExpertLens Works

ExpertLens can be implemented in two ways. The more traditional format involves up to four rounds of iterative elicitation and discussion. However, it can also be used to conduct real-time or "roundless" Delphi studies.

The Traditional, Round-Based Process

flowchart of ExpertLens process

flowchart of ExpertLens process

ExpertLens is based on the notions of iterative data collection and group feedback.

Round 0: Idea Generation (Optional)

Click to enlarge.

A sample of participants answers a series of open-ended questions or engages in an online discussion of issues that can help better understand a problem, help develop close-ended questions and/or answer choices, or design hypothetical scenarios used in subsequent rounds.

Round 1: Assessment

In Round 1, participants answer rating or ranking questions and explain their numeric responses.

In Round 1, participants answer rating or ranking questions and can provide feedback using open text boxes. Often, Round 1 questions are based on the results of Round 0. For example, the statements that participants are asked to rate or the scenarios that they are asked to evaluate may be based on data collected in the previous Round.


Round 2: Feedback and Discussion

All Round 1 data are automatically analyzed before Round 2 begins. Group and individual answers are presented in an easy-to-understand format.

Besides being a data collection tool, ExpertLens has a built-in data analysis capability. All Round 1 data are automatically analyzed before Round 2 begins. Means or medians, as well as quartiles, are calculated for each question. Study participants can not only see how the group answered Round 1 questions, but also compare their own answers to that of the group. Group and individual answers are presented in a table and in a graph format.


The analytic approach described in the RAND/UCLA Appropriateness Method User’s Manual is used to automatically measure group agreement.

The analytic approach described in the RAND/UCLA Appropriateness Method User’s Manual could be used to automatically identify areas of agreement and disagreement. Results of these analyses are presented to participants both graphically and in plain English.


Participants are also invited to participate in asynchronous and anonymous online discussion of their perspectives.

Participants are also invited to participate in asynchronous and anonymous online discussion of their perspectives. Discussion moderators can ask participants to explain why their answers might have been different from that of others and further clarify their perspectives.


Round 3: Reassessment

Finally, in Round 3, participants modify their original answers based on group feedback and discussion.

Finally, in Round 3, participants modify their original answers based on group feedback and discussion and rate their experiences using ExpertLens.

Real-Time Delphi

ExpertLens can also be used to conduct real-time or roundless Delphi studies. In such studies, participants see the group responses as soon as they answer a question. They can revise their original answer right away if they wish to do so. The main benefit of real-time Delphi panels is that they do not require participants to participate in multiple rounds.

RAND/PPMD Patient-Centeredness Method

ExpertLens can also be used to implement the RAND/PPMD Patient-Centeredness Method (RPM), a process for engaging patients and caregivers in clinical guideline development.