This paper shares lessons learnt from engaging multi-disciplinary practitioners in a complex field of social policy in a consensus building exercise using an adapted Delphi-method. It is written for organisations or researchers considering a similar approach. Suggestions include using participants' own language in subsequent surveys; minimising selection criteria; and mitigating the required time commitment of both researchers and participants.

Engaging multi-disciplinary practitioners in a complex field of social policy

A methodological discussion paper

Tom Ling, Michaela Bruckmayer

ResearchPublished Oct 21, 2021

The objective of this paper is to share lessons learnt from engaging multi-disciplinary practitioners in a complex field of social policy in a consensus building exercise using an adapted Delphi-method. Multi-disciplinary practitioners include all professionals, policymakers and researchers who can affect, or are affected by, the success of the organisation or service in question. This methodology paper discusses just one very focused component of the challenge of engaging multi-disciplinary practitioners: how practitioners from a wide range of backgrounds can be engaged through surveys. The issue of concern in the survey we conducted was Adverse Childhood Experiences (ACEs). This report focuses solely on the methods used, and is written for other organisations or researchers considering a similar approach. Suggestions include using participants' own language in subsequent surveys; minimising selection criteria of participants; and taking practical steps to manage potential weaknesses in the Delphi method such as the required time commitment of both researchers and participants.

Recommendations

Using participants' own language to provide a level playing field of terminology and perspective

  • Different stakeholders use different words and concepts to describe their world. We wanted to avoid using any unintended bias by using our own terminology and so opted as far as practicable to use the vocabulary provided to us by participants responding to the survey. Despite these efforts a small number appeared to be uncomfortable with the language we used in the second and third stages of the Delphi. However, it was also clear that groups with different 'situated knowledge' could enter inter-disciplinary conversations and we concluded that Delphis could work in such an environment, if handled with sensitivity and care by those developing the survey.

Minimising selection criteria to ensure engagement with a diverse range of practitioners

  • Working towards achieving consensus among experts requires deciding who these experts are and what exactly makes them an 'expert'. A person's expertise is commonly evaluated based on their qualifications, publication record, and reputation in the field. The strength of the Delphi method has always been pragmatic rather than epistemological. We avoided anything like a minimum education requirement and preferred to take advantage of our client's knowledge to identify those with a known engagement with (and commitment to) this topic. This approach achieved a meaningful debate and dialogue among participants but illustrated an inherent tension within the Delphi method.

Taking practical steps to manage potential weaknesses in the Delphi method

  • Despite its advantages, participating in a Delphi can be time-consuming—for both the participants and the researchers. To help make the process more manageable, we grouped the more than 200 qualitative responses into nine themes. Within these themes we tried to cover all suggestions we had received, selecting the words and phrases provided by participants that best conveyed the sentiments of their suggestions. In the next survey, rather than asking participants to vote again on all statements again, we focused on those areas where disagreement remained.

Topics

Document Details

Citation

Chicago Manual of Style

Ling, Tom and Michaela Bruckmayer, Engaging multi-disciplinary practitioners in a complex field of social policy: A methodological discussion paper. Santa Monica, CA: RAND Corporation, 2021. https://www.rand.org/pubs/research_reports/RRA1111-1.html.
BibTeX RIS

Research conducted by

This research was commissioned by The Early Intervention Foundation and conducted by RAND Europe.

This publication is part of the RAND research report series. Research reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND research reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.