Data Analysis Options

ExpertLens data can be analyzed in two ways.

A descriptive analysis can be performed to:

• Identify group responses.
• Estimate changes in group responses between rounds.
• Identify a degree of consensus among participants and groups of participants.
• Determine points of agreement and disagreement using the analytic approach described in the RAND/UCLA Appropriateness Method User’s Manual.
• Explore agreement between the decisions made by several concurrently administered panels.

For a more detailed discussion of descriptive analysis of ExpertLens data, see "ExpertLens: A system for eliciting opinions from a large pool of non-collocated experts with diverse knowledge," published in the journal Technological Forecasting and Social Change, and "Conducting online expert panels: A feasibility and experimental replicability study," published in the journal BMC Medical Research Methodology.

For an example of how to combine the results from two separate panels, see "On using ethical principles of community-engaged research in translational science," published in Translational Research.

A Bayesian approach to data modeling can be used to:

• Uncover potential reasons for disagreement.
• Track changes in individual responses.
• Identify any patterns of changes in individual and group answers.

For more details on ExpertLens data modeling, see "Collaborative Learning Framework for Online Stakeholder Engagement," published in the journal Health Expectations.

Qualitative comment analysis is a powerful yet often underutilized approach in the analysis of Delphi panel data. An important distinctive feature of the ExpertLens methodology is its capacity to combine quantitative and qualitative data, which helps qualitatively explain why participants answered numeric questions in a particular way and identify potential predictors of answer changes between Round 1 and Round 3. A thematic analysis of Round 2 discussion data can help uncover the reasons why participants changed their answers after Round 1. To expedite qualitative data analysis, ExpertLens uses artificial intelligence.

For examples of how qualitative data can be analyzed, see "On using ethical principles of community-engaged research in translational science," published in Translational Research, and "Patient engagement in the process of planning and designing outpatient care improvements at the Veterans Administration Health‐care System: findings from an online expert panel," published in Health Expectations.