UKRI Future Leaders Fellowships Evaluation Framework Report

The development and retention of research talent is key to the delivery of a wide range of UK Government strategies and commitments, including the Industrial Strategy, the Plan for Growth, the Life Sciences Vision, and the UK Innovation Strategy. The UK Research and Development (R&D) Roadmap and the R&D People and Culture Strategy articulate the UK's goals surrounding support for researcher careers and capabilities and the positioning of the UK as a world leader in attracting and retaining the best talent. UKRI, as the UK's largest funder of R&D, is central to achieving the outcomes set out in these strategies,with the Future Leaders Fellowship (FLF) programme as a key mechanism for delivery on these goals. The FLF, a £900 million investment, occupies a unique position within the landscape of early-career researcher programmes in the UK, supporting both UK and international researchers working in a wide range of disciplines and sectors, and encouraging novel and interdisciplinary research projects.

In June 2022, UKRI commissioned RAND Europe to conduct an evaluation of the FLF programme. The aims of this evaluation are to understand the impact of FLF, draw lessons regarding how programme implementation has enabled impact, and explore the programme's value for money. This report (the first project deliverable) presents the evaluation framework for the evaluation of the FLF. The report has been prepared to guide the evaluation and for UKRI to approve the planned approach prior to commencement of the core evaluation data collection and analysis tasks. The proposed evaluation approach is underpinned by two primary analytical frames: the FLF Theory of Change and an adapted version of the Kirkpatrick model, a framework traditionally used for evaluating training interventions. Our evaluation approach will comprise: a process evaluation; an impact evaluation comprising a combination of quasi-experimental design and theory-based evaluation; and an economic evaluation (subject to further feasibility assessment). The process evaluation will aid understanding of how programme design, delivery and support has been received, whether the programme targets have been met and identify key challenges or barriers, if any, thereby identifying opportunities for learning. It will also assess whether the FLF has been designed and implemented in a way that supports the anticipated impacts of the programme.

The quasi-experimental impact evaluation will aid in assessing the extent to which selective outcomes have been achieved by FLF fellows compared to a proposed counterfactual group. This comparative assessment will be supplemented by a crucial theory-based evaluation which will capture more qualitative outputs and outcomes and explore whether the FLF is on track to achieve sustained longer-term impact. The theory-based component will include a realist analysis of the context and the mechanisms through which a given set of outcomes have been achieved for the fellows. This approach has the advantage of not only providing evidence of whether FLF has generated impact but also of how it has done so and for whom, so that the programme and others like it can be further improved in the future. An economic feasibility assessment, to be undertaken in parallel with the impact evaluation, will focus on assessing the viability of undertaking a cost benefit analysis and exploring mechanisms for assessing value and effectiveness.