The Research Excellence Framework (REF) is a mechanism to allocate funding to universities for the research they carry out. In 2014, the research of 154 UK universities was assessed, accounting for the efforts of 52,061 academic staff members. For the first time, the impact that the research had on wider society was part of the assessment. Researchers' output and the research environment were also measured.
Impact was assessed using case studies that had reported a selection of impacts that had occurred since 2008 as a result of university research. The academic departments' previous and planned impact strategies were also examined.
Over the past two years, RAND Europe has evaluated the submission process used by universities as well as the process of assessing research by academics and research users. Our research aimed to better understand the benefits, challenges, and consequences of the process—and suggest improvements that could be made for future rounds of assessment.
We found that as a result of the inclusion of the impact element, visible cultural change occurred at the institutional level. Examples include: many universities setting out an impact strategy, the inclusion of impact as a criterion for promotion of academic staff, implementing systems to capture and store impact, and changing the employment status of fixed-term posts to permanent for impact officers. Our evaluation also showed a cultural change at the individual level, reflected in the way that academics, who were involved in preparing impact case studies and templates, now approach their research. Early recognition of factors leading to impact has become a greater part of their research. This expanded approach is evident in how researchers engage with relevant stakeholders in the design, as well as the results of a project, and in how researchers produce documents tailored to different audiences.
In addition to noting these changes to behaviour, our study found a number of negative, perhaps unintended, consequences of assessing impact.
In submitting impact case studies, universities had to align them with the REF definition and provide sufficient evidence to back up the claims made. As a result, many universities provided examples of impact that were not valid for evaluation purposes, such as when the impact was commercially sensitive or resulted in changes in attitude that were harder to measure. As a result of these constraints, the impact that was assessed may not have been representative of the total impact of research in the university sector.
There is also concern from the university sector that the assessment of impact may begin to undermine more theoretical or “blue skies” research. As universities seek to attract funding, the focus of research may shift toward more applied questions, and toward areas that can more easily demonstrate impact in the short term. The question (and inherent challenge when seeking an answer) now is: What is the correct balance of basic vs. more applied research?
Overall, the UK has shown it is possible to assess research impact and allocate funding based on the results of the assessment. Since this is the first time the assessment has been conducted, and on such as large scale, the effort should be recognised and applauded. As we start to approach the next assessment period, there is opportunity for discussion across the university sector to tweak and refine future assessments of impact. While RAND Europe's research shows a range of consequences in assessing impact in the short term, determining longer-term consequences will require a greater understanding of the types and mix of research funded.
Catriona Manville is a senior analyst at RAND Europe based in Cambridge.
This commentary originally appeared on Open Forum Events on July 6, 2015. Commentary gives RAND researchers a platform to convey insights based on their professional expertise and often on their peer-reviewed research and analysis.
