Evaluating the Assessment Process for the Impact Element of Research Excellence Framework 2014

Background

The Research Excellence Framework (REF) 2014 was a nationwide initiative to assess the quality of research in UK higher education institutions (HEIs). For the first time, REF 2014 introduced the wider impact of research, alongside the quality of research and the vitality of the research environment, into the assessment of research excellence in UK HEIs. This wider impact of research was weighted at 20% of the total assessment for 2014. Understanding how the impact element of the REF 2014 submission process was assessed will be important for future rounds of assessment, to ensure fairness to all parties and to prevent unnecessary burden.

Goals

Following our evaluation of the submission process, the four UK higher education funding bodies commissioned RAND Europe to evaluate the assessment process for the impact element of the REF 2014. The objectives of this phase of the evaluation were to:

• Explore the strengths and weaknesses of the assessment process in delivering reliable, robust and fair outcomes.
• Identify any unforeseen issues emerging during the assessment process, and evaluate the management of these.
• Outline broad implications for the assessment process for impact in future REF exercises.
• Understand the findings of this study within the context of the evaluation of the submission process.

A mixed-methods approach was used to evaluate the process including face-to-face and telephone interviews and an online survey with members of the assessment panels and a cost analysis to produce robust conclusions and recommendations.

Key Findings

• By a large majority, the academics and research users on the panels felt that the process enabled them to assess impact in a fair, reliable and robust way.
• Bringing together different perspectives of academics and research users was seen to be successful and valuable.
• We identified incremental improvements and areas for discussion that can inform subsequent REF exercises and other countries planning to run similar systems. These include how to manage variations in the way the process was conducted; how to avoid the risk of unsubstantiated and false claims being made; and how to clarify the processes for assessing different kinds of impact.

Publication

Project Team

Catriona Manville
Tom Ling
Susan Guthrie
Bryn Garrod
Sonia Sousa
Sophie Castle Clarke
Marie-Louise Henham
Anne Kirtley

---