The National Institute for Health Research Leadership Programme

Background

This article presents the findings of an evaluation of the National Institute for Health Research (NIHR) Leadership Programme conducted by RAND Europe. The aim of the evaluation is to assess the extent to which the NIHR leadership programme has delivered on its three core objectives of developing individual leaders, building research team capacity and fostering leadership in the wider research community, and to look at the impact of the programme more broadly. There are four key aims to the evaluation:

1. to evaluate the programme's activities against its stated objectives, including whether the methods of the leadership programme map against the objectives;
2. to determine the impact of the programme in relation to participants' leadership skills and participants' impact on immediate staff and colleagues;
3. to evaluate the extent to which leadership skills and capabilities are becoming “embedded” within NIHR as a result of the programme; and
4. to determine the contribution of the programme to NIHR and health research in general, and to begin to understand the feasibility of evaluating the impacts of the programme going forward.

The evaluation is designed to enable the English Department of Health (DH) to account for the expenditure of public funds and extract lessons for the future, as well as develop plans for the next phase of the leadership programme. The focus and methodology of the evaluation has recognised these considerations.

The NIHR Leadership Programme began delivering to researchers in January 2009. It was commissioned against a backdrop of an increasing emphasis on high-quality clinical research in the National Health Service (NHS) in the wake of Best Research for Best Health (Department of Health, 2006) and the Cooksey Report (Department of Health and Davies, 2006), and a need to deliver high-quality research within the NHS.

Delivery of this agenda requires an effective research base and system of leadership supporting it. It is widely accepted that leadership training can have a hugely beneficial effect on an organisation, no matter whether its setting is in research, the public sector or private industry. Therefore, there was a real opportunity to develop the skills and capabilities of NIHR leaders and to enable them to make a real difference to the health research environments in which they work. In this context, the programme can be thought of as an attempt at a “science policy intervention”: by investing in the leadership skills of senior researchers and future developing leaders within NIHR, there might be an improvement in the ability of the research base to deliver a strong, robust and globally competitive UK health research sector.

The National Institute for Health Research Leadership Programme

By focusing on three key objectives of developing individual leaders, building research team capacity and fostering leadership in the wider research community, the programme aims to help achieve change by first enabling researchers to become leaders, then supporting them in achievement of wider institutional, organisational and national goals. There are three levels of participant in the NIHR Leadership Programme:

1. senior leaders are the most senior researchers in the NIHR;
2. development leaders are individuals who are on course to reach senior positions; and
3. trainee leaders are making the transition to being independent researchers and are taking on their first significant management and leadership roles.

The NIHR Leadership Programme is run by the Ashridge Business School (hereafter, Ashridge) and comprises four streams of activity:

1. development in the work setting;
2. leadership practice;
3. a combination of one-to-one work and support, especially for more senior leaders, and groupwork and support, particularly for developing leaders; and
4. provision of focused skills relevant to leadership in a research setting.

Within these four overarching streams a range of specific activities or “interventions” are offered to participants according to their leadership group. These include:

• accompanying and one-to-one coaching
• biannual learning conferences
• bespoke 360-degree feedback
• virtual workshops
• tailored learning guides
• periodic phone conversations and conferences on emerging issues
• biomedical research unit or biomedical research centre strategy workshops
• action learning or peer project support groups.

Each programme is tailored to the leadership cohort level and the individual being supported, although there are some common features shared across all leadership groups. Senior leaders receive greater accompanying and individualised coaching support and are offered an “à la carte” approach to the programme activities, while trainee leaders follow a more structured programme centred on group learning and targeted workshops. The action learning groups are common to all levels of leader and are meant to be one of the main vehicles for participant learning during the programme, where insights and learning on the programme are converted into actionable solutions to real-world problems.

The Evaluation Process

In order to evaluate the impact of the leadership programme and the extent to which it is delivering against its three objectives, we used the Kirkpatrick Model for evaluation.* The Kirkpatrick Model uses four levels of analysis to understand how the learning was received and the wider impacts that it had on the participant. For our evaluation, the model was applied to understand basic reactions to the delivery of the leadership programme; what learning and new skills were acquired during the programme; whether any behaviour change had occurred as a result of the programme; and finally, whether the leadership programme was contributing to wider outcomes and results. This framework was applied throughout the evaluation methodology, which consisted of five primary workstreams.

• Desk-based research to understand the approach and philosophy of the programme, review the findings of internal evaluations and identify literature on leadership evaluation.
• A web-based survey of all programme participants to solicit as large a range of views as possible about the impact of the programme.
• A series of semi-structured, in-depth interviews with a selection of leaders and their colleagues to develop a deeper understanding of the specific impacts and areas of learning on the programme.
• Structured benchmarking to enable comparative analysis against programmes addressing similar challenges and to allow us to make informed judgements about effectiveness of the NIHR Leadership Programme in context.
• A small workshop with health and research leadership experts to test the robustness of the findings and recommendations.

In any evaluation of an intervention such as the leadership programme, there will be a problem of attribution and contribution. In other words, while the leadership programme may contribute to wider outcomes, we may not be able to attribute these outcomes solely to the leadership programme. While we have taken steps to link outputs and outcomes to specific inputs and processes (through careful design of survey and interview tools, and triangulation of evidence), we recognise that there are caveats to this approach.

Evaluation Findings

• Overall, the NIHR Leadership Programme delivered by Ashridge is positively received by participants. The majority of interventions that form part of the programme are perceived by participants to be useful, and they would recommend the programme to others. However, there are differences of opinion both within and across levels about the usefulness of specific interventions within the programme.

The leadership programme fills an important gap in academic and clinical researchers' professional development which otherwise might not be filled through their own institutions. We believe that this attests to the overall positive contribution that the programme makes to the sector, but also should be a caveat to consider when reviewing the findings. This is because individuals have little against which to compare the programme.

While all levels of leaders found the one-to-one coaching activities useful, development and senior leaders did not find other activities as useful. Overall, action learning groups were thought to be least useful by these two groups of researchers and there was variable uptake, with many not participating in them. This is surprising, given that the action learning group is a forum for bringing together the learning from all other leadership programme activities.

Trainee leaders found the action learning groups and the tailored workshops to be the most useful; however, generally they indicated a wide range of activities to be useful and programme coordination was viewed most positively by this group (as compared to some less positive views from development and senior leaders). Trainee leaders receive the most structured delivery of the activities, and we conclude that their more positive reflections on the coordination of the programme as a whole are due in part to this.

• There has been a particularly strong contribution to individual leadership development, personal awareness-raising and confidence-building as a result of the programme. There is some evidence of the programme contributing to the objective of building research teams and institutional leadership.

The majority of participants across all levels indicated that the programme had either a “major” or “significant” impact on their personal approach to leadership. The participants indicated that they were acquiring new skills and that these were translating, to varying extents, into changed behaviours.

Although individuals in fact may be learning and adopting new behaviours which could have wider institutional impacts, they may not always be aware of the link between the two. More than 75 percent of the respondents across all groups indicated that they are better able to manage their current institutional tasks and are more aware of the external context in which they operate. This suggests that there may be impacts on their institution due to the fact they are more aware of the context in which they work and how to manage their tasks better. However, fewer than 50 percent of development and trainee leaders thought that the programme helped them to have a major or significant impact on their own institution, and only 53 percent of senior leaders felt that they had a major or significant impact on their institution. In their comments, many respondents indicated that the impacts they felt they were having were related to being more reflective and confident as a leader, again emphasising the importance that the programme has had in fostering personal leadership development, but raising questions as to how higher-level objectives were being met.

• Although individuals are reporting strengthened leadership characteristics and leadership performance as a result of the programme, we did not find evidence that research outputs and research performance are improving, and the links between leadership and research performance could be strengthened.

There is weaker evidence for the impacts on research leadership, specifically within the NIHR, or more broadly within the biomedical research sector. Participants across all leadership cohorts indicated that the programme had helped to improve their performance as a leader significantly by building their confidence, but it had not been as significant in building their credibility as an intellectual leader or in helping them to influence colleagues. Insights from the in-depth interviews suggest that improvements in leadership were more about interpersonal relationships and self-awareness, rather than strengthening their ability to overcome or address research challenges: in other words, to contribute to strengthened research capacity in the NIHR.

We include a caveat in this next point, in that higher-level impacts such as improved research outputs or higher quality research are of a more long-term nature, and we would not expect to find strong evidence of these only three years into the programme. More than 60 percent of all participants, including more than 75 percent of senior leaders, reported that they believe the programme has helped them to strengthen links across research disciplines with other colleagues. However, when asked about the extent to which individuals felt the programme was enabling them to have either a personal, institutional or wider research community impact, many commented on improved self-awareness; few mentioned improved ways of doing research which otherwise might be suggested by the previous finding. This suggests that the link is not yet being made between improved leadership and research performance, and the individual's leadership role in the wider community.

• The delivery of the NIHR Leadership Programme is currently balanced in a way in which there is more emphasis on individual leadership development and less emphasis on the development of institutional and wider research community leadership within the NIHR. This suggests that there is either a mismatch in the way that the programme interventions are linked to the wider objectives and aims of how the NIHR would like to foster and support its leaders, or it may be a matter of timing and that the future focus of the programme should be balanced more equitably.

The Venn diagram in Figure 1 illustrates what we believe the current balance of the NIHR Leadership Programme to be across the three objectives.

Here, each core objective of the leadership programme is represented as a circle, with the overlaps between them reflecting how the objectives mutually support each other and how each overlap has a corresponding set of outcomes. For example, overlaps between activities which support personal and team leadership and those which aim at the objective of fostering wider NIHR leadership might be expected to reflect individual and collective contributions to the leadership of research in the NIHR. Equally, overlaps between institutional and NIHR leadership might lead to outcomes that foster connections between institutional research strategy and the wider NIHR research strategy. It should be asked whether this model of leadership is the right model for NIHR, and whether this will allow it to achieve its long-term aims of transforming biomedical and health research in the UK.

Evaluation Recommendations

• We recommend that Ashridge needs to take a holistic look at the programme that it has developed and re-evaluate whether it is fit for purpose across each leadership cohort; also, whether it is aligned and balanced with overarching NIHR objectives.

In light of the findings about the usefulness and integration of learning activities in particular, we believe that the structure of the programme for each leadership cohort needs to be critically examined in order to ensure that the activities are delivered in an integrated and coordinated fashion that clearly supports wider learning objectives. It seems that there is a particular need to address integration of programme interventions for senior leaders, as opposed to the current model where they are able to choose individual interventions in which they are interested, or for which they have time. For example, by requiring more structured participation in the programme, as opposed to the current “à la carte” model, there might be more opportunities to ensure that programme learning objectives are coordinated and achieved more readily. In addition, action learning groups, or a similar method whereby integration of programme learning occurs, should be more firmly established in the higher programme levels and steps taken to ensure that they are playing the role that they need to be in the wider programme architecture.

In order to help participants make the link between research performance and leadership, we recommend that Ashridge reconsiders the team of coaches and facilitators that deliver the programme and bring in specific experience of academic, biomedical or health research leadership expertise at key stages in participants' development over the course of the programme. This will enable participants to learn and connect leadership concepts to research challenges from people who are familiar with the research environment and contexts in which the participants function. In addition, Ashridge might consider alternate structures to the programme, where more active steps are taken to encourage both the horizontal and vertical integration of researchers and leadership levels, so that individuals learn from those facing similar disciplinary challenges but perhaps different management or institutional challenges. To this end, Ashridge might look to other leadership programme models, for example the United States National Institutes of Health (NIH), which aims to foster vertical and horizontal integration across individual research institutions through careful selection of programme participants for each leadership course.

• We recommend that Ashridge addresses the “end” of leadership and how individuals leave the active programme, yet remain integrated in a wider leadership community.

We note in our evaluation that currently there is no formal system in place for “rolling off” the programme if one is a senior or development leader. While the trainee leader programme has a formal graduation ceremony at the end of the 18-month programme, the endpoint for the other two programmes is not as clearly defined. We found little evidence that this aspect had been appropriately addressed for these higher levels, and a more holistic consideration of the beginning, middle and end of each programme is needed.

Without a clear beginning and end for the programme, it can be difficult for participants to fully understand and appreciate the wider context in which the leadership programme is aiming to achieve change. Moreover, it can be difficult to discuss the continuity, future skills development and continued learning needed to sustain a leadership community. It is difficult to see how a wider leadership community is being established without this aspect being addressed, and in fact many participants felt that this was actively discouraged. Therefore, there is a missed opportunity in developing a wider community of leaders within the NIHR. This would contribute to individuals taking more responsibility for their position in the wider research community, and would be likely to contribute to better achievement of the third objective of the programme. In addition, it would help to ensure that the objectives of the programme as a whole are met and are contributing to wider NIHR objectives.

• We believe that there is an issue in the way that the leadership programme connects to a wider theory of change for the NIHR. We suggest that if the DH would like to see the NIHR Leadership Programme as a science policy intervention, there needs to be a shift towards better integration and more equal balance across the three leadership objectives, and greater attention to the specific outcomes that are desired from the overlap of each.

We believe that this is an issue for the DH and senior leadership of NIHR to consider and work through with Ashridge. There is an opportunity to understand better how an NIHR theory of change in the biomedical and health research sector can be achieved through delivery of the leadership programme.

• We suggest that senior leadership within the DH might consider a rebalancing in the programme in the context of the following proposition: leadership should be at the core of NIHR science policy.

This would not only require continued and ongoing investment in the leadership programme, but concentrated efforts to embed leadership development and awareness throughout the biomedical and health research landscape. This might be achieved, for example, by requiring all grant applications to have a “leadership and development” section. It might involve more concentrated efforts at enabling a group of NIHR leaders to work within their research communities to lead on particular research issues, public health challenges or shared academic concerns, such as the opportunities and challenges provided by the Research Excellence Framework. Ashridge could facilitate and support these community-driven initiatives through the leadership programme and provide individuals with the leadership guidance and skills to do this.

On the basis of all the above, we conclude that Ashridge has built up a suitable foundational knowledge of the NIHR and has an awareness of the individual leadership needs of researchers. After three years, we believe that it has achieved an acceptable level of success in delivering strongly on the first objective, and is beginning to deliver on the second objective. The third objective will require more time to be met, but could be more readily achieved through consideration of our recommendations.

References

Department of Health (Research and Development Directorate), Best Research for Best Health: A New National Health Research Strategy, 2006. As of October 12, 2011:
http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_4127127

Department of Health and S. C. Davies, Sir David Cooksey's Review of UK Health Research, 2006. As of October 12, 2011:
http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_062915

Note

* See http://www.kirkpatrickpartners.com/OurPhilosophy/tabid/66/Default.aspx.