It Worked: REF 2014 Was Successful in Assessing Research Impacts

RAND Europe's evaluations of the impact element of the UK's Research Excellence Framework 2014 show that researchers can describe the wider societal benefits of their research, and that these impacts can be assessed and compared.

Research outputs from universities are assessed every five years to determine future funding allocations from government. In 2014, for the first time, the Research Excellence Framework (REF) included an assessment of research impact. This component was worth 20 per cent of the score awarded to each institution.

RAND Europe has conducted two evaluations of the impact component of REF 2014 that will inform future, similar exercises. The first study evaluated universities' preparation of submissions describing impacts, in the form of case studies and impact strategies. The second study examined the way in which submissions were assessed by panels of academics and research users.

Our findings on the process of preparing impact submissions:

• Academics and institutions reported a number of benefits of engagement with the impact component of REF 2014, such as developing the ability to identify and understand impact, and the stimulation of broader strategic thinking about impact.
• Complying with the impact component of the REF imposed a burden on resources and time for individuals and for the sector as whole. Academics perceived the process to be overly burdensome to research users who were involved to provide evidence, but this was not the experience of research users.
• A cultural change appears to be taking place whereby institutions and individual academics are adopting a new focus on the current and potential impacts of their research. Alongside a wider impact agenda, REF 2014 has contributed to this shift.

Our findings on the assessment of evidence of impact:

• By a large majority, the academics and research users on the panels felt that the process enabled them to assess impact in a fair, reliable and robust way.
• Bringing together different perspectives of academics and research users was seen to be successful and valuable.
• We identified incremental improvements and areas for discussion that can inform subsequent REF exercises and other countries planning to run similar systems. These include how to manage variations in the way the process was conducted; how to avoid the risk of unsubstantiated and false claims being made; and how to clarify the processes for assessing different kinds of impact.

Catriona Manville, senior analyst and lead author on the evaluation reports, said: “Given that the process for assessing impact was new, our evaluation shows that it worked, and it worked well. The academics and research users who carried out the assessments felt that the process was fair. REF 2014 now provides a valuable working model for measuring impact in other countries and research systems.”

# # # #

RAND Europe is an independent not-for-profit research institute whose mission is to help improve policy and decisionmaking through research and analysis.

Project background: The evaluations were performed by RAND Europe for the UK higher education funding bodies.

Find more project information at: www.randeurope.org/REF2014impact