Leadership as a Health Research Policy Intervention

Background and Context

High quality and high impact research requires both a highly skilled researcher base and a system of leadership supporting it. The National Institute for Health Research (NIHR) Leadership Programme was established in 2009, and can be thought of as a science policy intervention to develop the leadership skills and capabilities of current and future NIHR researchers. This is expected to contribute to research performance and impact, and ultimately to benefit patients.

RAND Europe evaluated the first phase of the programme (2009–2011). We found it was well received and had a substantial impact on participants' personal approach to leadership. It had also helped build effective research teams, but had not had much impact at institutional and system levels at the time. The NIHR Leadership Programme was re-commissioned and the second phase of the programme began in 2012.

Through enhancing individual leadership skills and capabilities and building a connected “community of practice” around applied and clinical researchers, the phase 2 programme (2012–2014) aimed to contribute to leadership capacity-strengthening at individual, institutional and system levels. The enhanced leadership capacity was, in turn, expected to facilitate a world-class research environment for better healthcare; help strengthen translational and applied research capacities; facilitate a greater degree of collaboration in the health research system; and help steer a social movement and foster a change in mindsets and attitudes about agenda-setting. The phase 2 programme included three main streams: NIHR Leaders, NIHR Trainees and R&D in Trusts. Continued engagement of alumni from previous cohorts was facilitated via the Leaders Stream.

Evaluation Aims

RAND Europe evaluated the second phase of the programme. We aimed to assess the programme against its core objectives and to capture progress towards them (including impact at individual, institutional and system levels); examine associated enablers, challenges and causal mechanisms at play; and explore scope for learning from other leadership programmes.

Study Design and Methods

Our evaluation was rooted in a theory of change, realist approach. A theory of change sets out the building blocks needed to deliver on a programme goal, through a pathway of interventions, and based on a range of assumptions about the underlying logic and types of interventions which can lead to desired results. Theories of change tend to be valued in programme planning and evaluation because they help create a shared view of what a programme's vision and strategy is; how it will be pursued; and what can be done to assist in identifying measures for capturing learning and reflecting on progress. As with the phase 1 evaluation, we referred to elements of the Kirkpatrick model of leadership evaluation in the process of developing the theory of change and evaluation framework. This model asks us to consider progress at four levels: basic reactions to programme, acquired learning and skills, behavioural change, and contributions to targeted outcomes.

The evaluation was implemented through a combination of workshop, survey and interview methods. The workshop helped develop the evaluation framework. The survey enabled breadth in the number of people who could contribute to the evaluation and in the diversity of issues we could explore. The interviews allowed us to explore emerging themes in more depth, and to investigate links between the programme design and its impacts, as well as associated enablers and challenges. We gathered evidence from multiple stakeholders to ensure a rounded evidence base and appropriate accountability. We also conducted a focused benchmarking exercise to examine how the NIHR Leadership Programme broadly compared to other leadership programmes across sectors.

It is important to note that the evidence presented in this report comes from interested parties—participants in the leadership programme and provider documentation. A detailed audit of the information is outside the scope of this project. Despite this caveat, the evaluation project team feels that an open and transparent rapport with programme stakeholders has been established and sustained. Triangulation across multiple sources of evidence, multiple methods and through time lends further confidence in the objectivity of the findings and recommendations.

Results

The evidence from our evaluation highlighted the following key outputs and impacts:

Motivations and expectations of participants: Across all streams, personal development opportunities were a key motivation for taking part in the programme. Prospects for networking, exchanging ideas and sharing experiences were also seen as important. Trainees and R&D in Trusts stream participants hoped to become more effective in establishing and managing research groups, and Leaders hoped to become more effective at an institutional level, for example in their influencing abilities.

Programme design: Overall, participants valued the activities that were undertaken as part of the second phase of the NIHR Leadership Programme, and felt that the design of the programme was generally appropriate. Action Learning Sets, networking opportunities and one-to-one coaching were seen as the most useful activities. Action Learning Sets enabled participants to seek advice from impartial peers on a diverse range of issues, and to exchange ideas from multidisciplinary experiences. The biggest benefits of one-to-one coaching were in helping develop individual confidence and in providing bespoke support around the challenges particular individuals faced.

Individual-level impacts: The programme helped develop a range of leadership skills important for individuals' career development and for wider organisational impacts. This includes skills that enable increased responsibility and career progression, the better management of research teams, enhanced self-awareness, reflective capacity and self-confidence. Strengthened collaboration skills were also seen as an important impact from the programme, and the programme's design created a vehicle for exposure to new potential collaborators, the establishment of informal relationships, the strengthening of existing relationships and changes in ways of collaborating. There was limited impact on participant ability to manage physical and financial resources in research teams, and future phases of the programme may wish to reflect on this as part of the institutional leadership capacity-building agenda.

Institutional-level impacts: The leadership programme also contributed to the establishment of new or strengthened institutional-level relationships, had an impact on an individual's leadership approach to staff training and development, and increased participants' awareness of what leadership means in an institutional context. Participants also felt that the programme enhanced the role they play in their institution's capacity to respond to structural change. The improvement intention within the R&D in Trusts scheme has a particularly notable institutional impact. It allowed managers and directors to work together on a concrete improvement task, provided an opportunity to put the acquired leadership skills into practice, and helped raise the profile of R&D within the institution.

The R&D in Trusts Stream also highlighted a particularly strong impact on improved relationships of institutional value, including relationships with members of other NHS trusts, and specifically with R&D counterparts. Only a few respondents (across streams) felt the programme had reduced duplication between NIHR and NHS units or led to formal collaborations between NHS service providers or between other NIHR units. Perceived barriers included the relatively rigid and hierarchical structure of health systems and universities, and the absence individuals who might be most challenged by structural change on the programme.

Systems-level impacts: The key types of systems-level impacts identified through our evaluation include: strengthening relationships within the NIHR community, an enhanced profile of leadership and its importance within the NIHR, a greater understanding of the NIHR and of the wider health system by participants, and new collaboration prospects with a wider set of health system stakeholders. Survey respondents across streams agreed that leadership positions in the NIHR now command greater respect. Participants also felt that the programme contributed to their understanding of NIHR's role within the wider health system, its priorities, aspirations and ways of working. Systems-level impacts were the weakest for the trainees scheme as participants in this scheme had not really had a sufficient level of interaction and engagement yet (e.g. with policymakers and wider stakeholders), given their career stage.

Continued engagement: The majority of alumni from the Leaders Stream who completed the survey felt there is merit in continued engagement with the NIHR Leadership Programme and that more opportunities for engagement, more advance notice of events and somewhat more flexibility in programme delivery (location, relevance of sessions) would act as enablers.

Discussion

Our evaluation shed light on a diversity of factors which have influenced the second phase of the NIHR Leadership Programme, and its impacts.

Key enablers included:

• Time and space to think: The NIHR Leadership Programme created a unique opportunity for self-reflection and the “space to think” away from the office, which were very important enablers of individual leadership skills development.
• Networking opportunities to share experiences, ideas and explore new prospects: The networking opportunities created by bringing together a broad range of participants and facilitating repeated interactions played an important role in raising awareness of the diversity of NIHR's goals, priorities and activities. A better connected NIHR community was, in turn, said to be conducive to the sustainability of the NIHR as a health and science policy institution.
• A concrete improvement task with expected organisational impact: The improvement intention in the R&D stream of the programme has enabled participants to articulate their intentions for organisational change and has helped raise the profile of R&D in their Trusts.

Key challenges included:

• Scope for further clarity on what the NIHR expected people to achieve by virtue of participating in the programme was seen as an obstacle to maximising potential impacts.
• Limited opportunities for sustaining engagement with the programme post-completion of core training are thought to be a barrier to further enhancing and nurturing the leadership skills and capacity that the programme is helping establish.
• Enhancing organisational and system-level impact: While the programme has made a significant contribution to the personal development of NIHR Leaders and shown some impacts and prospects for further impact on institutions and the wider system, enhancing the scale and scope of such effects remains a challenge.

Our benchmarking exercise provides a range of cross-cutting insights on mechanisms used to facilitate different levels of impact in other leadership programmes in the UK and internationally. These may be helpful for the NIHR to consult when thinking about the next phase of the programme. For example:

• Some of the ways by which system-level leadership capacity building was pursued in our benchmark examples included impact groups and challenge projects focused on system-level issues in a sector. These created a practical and formal way of working as a leadership community.
• Institutional-level leadership capacity-strengthening in our comparator programmes was facilitated through a range of interventions, including: cross-departmental and cross-disciplinary team work between members of an institution; combining taught and experiential learning with the anchoring of newly acquired skills on the job (e.g. in clinical duties); projects to design, implement and evaluate organisational improvement intentions; and formal line-manager support and engagement with a leadership programme.
• Individual-level leadership capacity building tended to be pursued through diverse psychometric tools and topic-based training, and in one instance through the formalisation of training course completion as a prerequisite for career progression.

Conclusion and Recommendations

There are a number of areas for policy consideration that emerge from the evaluation evidence. These relate to NIHR Leadership Programme design, facilitating impacts, continued engagement and sustainability, and evaluation. We hope that these will be helpful in framing future phases of the programme and its delivery:

1. The NIHR and training provider should consider making the relationship between activities proposed for the next phase of the leadership programme and each dimension of leadership capacity (individual, institutional and system) more explicit.
2. The NIHR may wish to consider making the relationship between the NIHR Leadership Programme and wider NIHR programme goals more explicit to participants.
3. Training provider knowledge of the health research sector and the challenges leaders in this sector face was important for participants. The NIHR should bear this in mind when selecting suppliers.
4. There is a need to reflect on selection criteria to the programme. These could consider individual motivations and needs, prospects for organisation and system impact and the overall mix within a cohort and across them.
5. The NIHR and training provider may wish to consider the scope for additional interaction across cohorts, streams and disciplines as a value-added activity. This could help tackle leadership challenges relating to silos in the system (such as effective multidisciplinary working, working across hierarchies).
6. There is scope to think creatively about new ways of facilitating organisational-level impacts, drawing on the experience of the NIHR Leadership Programme and other leadership programmes in the system. For example, the NIHR may wish to consider what the most appropriate mix of participants in teams might be, and whether there is scope for some collective leadership training interventions that bring participants from the same organisation together. The organisational improvement intention might be valuable across programme streams. Ways to ensure line-manager engagement with the training programme, especially for trainee schemes, could also be worth considering.
7. The NIHR could consider new ways of facilitating systems-level impacts (e.g. some examples might include a leaders task force or working group on systems-level challenges, or improvement projects targeted at systems-wide issues, as part of the Strategic Collaboration Initiative).
8. The NIHR should consider ways to diffuse leadership skills into the wider research system (e.g.training the trainer approaches, where NIHR Leadership Programme alumni could facilitate leadership capacity building activities in the wider health research system, for example as facilitators of Action Learning Sets).
9. Consider ways to keep alumni engaged with leadership capacity-strengthening activities for their individual benefit and the benefit of the wider health research system (to ensure a connected community of empowered leaders).
10. Continue evaluating the programme so that adaptation and learning could feed into continual improvement, and ensure accountability. It may also be worth tracking organisational and systems-level impacts from alumni over time (as these can take time to materialise), through targeted and brief thematic evaluation.

In addition to the policy recommendations stemming from the evaluators analysis of evidence, there were some additional recommendations stemming directly from participants. These included: increasing the visibility of the programme across the NIHR; addressing operational challenges relating to the planning, location and timing of events; strengthening incentives for participation of R&D directors; considering scope for further exposure to leadership theory, and exploring prospects for a more structured approach to networking activities.

These reflections are also important to consider in the context of leadership programmes as science policy interventions. As we have shown through this evaluation, the following are important for informing policy decisions in this space: relationship building and networking; bespoke approaches to the needs of individuals and groups, at specific stages of their career pathways and across them; collaborative approaches to joint working; and evaluation mechanisms which can explore causation and relate the programme design and implementation to diverse desired impacts. As a policy intervention, the NIHR Leadership Programme has strong potential to identify and nurture outstanding leaders who can span the boundaries of their individual, organisational and systems-level professional identities, and consider their roles in the context of wider systems-level ambitions for pursuing research excellence, nurturing leadership and research capacity across the system, and helping their research to be disseminated, ultimately for patient benefit.

We hope that the contents of this report will be helpful to the NIHR, as it frames and implements the next phase of the NIHR Leadership Programme.

This is an independent study commissioned and funded by the Policy Research Programme in the Department of Health and conducted by RAND Europe.