Understanding researcher views on the UK’s national assessment programme, REF 2021

A real-time evaluation of the REF 2021 assessment process found that overall views on the REF are mixed. Many researchers perceive that the REF has a negative influence on UK researchers and the research community, but views on specific aspects and influences of the REF are more nuanced.

New Africa/Adobe Stock

What is the issue?

The Research Excellence Framework (REF) assesses the quality and wider impact of research in the UK’s higher education institutions (HEIs), as well as the research environment in which it is conducted. The assessment then guides how funding by the UK’s research funding bodies is allocated.

The REF is associated with driving research quality and raising the profile of research activities within institutions. Despite its importance in shaping research cultures, there is little systematic and nuanced evidence about how academics across the sector view the REF, and which aspects are viewed favourably or unfavourably.

How did we help?

The REF 2021 took place from 2014 to 31 March 2021. RAND Europe, in collaboration with the University of Sheffield and Cardiff University, led a real-time evaluation of the process while researchers submitted their work, collecting data between October 2020 and January 2021.

The aim was to understand researcher perceptions around the REF, how they viewed the experience and to see how it affects academic environments. The project team also explored how REF policies and changes are embedded in the way submissions are prepared and delivered. To provide an analysis on how views have changed over time, the study built on the experience of a 2019 pilot exercise.

RAND Europe analysts conducted two online surveys of 3,080 researchers — a national survey of researchers across the UK and a longitudinal survey of researchers at four UK HEIs involved in the pilot study. Online focus groups were held with research managers and researchers, and one-to-one interviews conducted with institutional leaders. Ultimately, individuals from a cross-section of senior and early-career researchers across a wide range of disciplines at 112 HEIs across the UK were engaged with the study.

This study was commissioned by UKRI, on behalf of Research England, the Higher Education Funding Council for Wales, the Scottish Funding Council and the Department for the Economy of Northern Ireland.

By holding the review in real time, alongside preparations for the exercise, the study findings can help accelerate the process of learning and revision post-REF 2021 to ensure that UK funding bodies continue to implement policy that is informed by evidence.

What were the key findings?

Proportional representation of survey respondents' views on the influence of the REF on the research community, with answers to the question "How has the REF influenced your research community in the UK in the past four years in the following areas?" The areas, from most increased to most decreased, are: gameplay, open research, quantity, public relevance, quality, interdisciplinarity, integrity, novelty, and authenticity.

Click to download the full infographic

The influence of the REF on researchers and the research community

Most researchers believe that the REF has increased open research (64 per cent), the public relevance of research (52 per cent), and the quantity of research (60 per cent) in the research community.
More than 85 per cent of researchers believe that the REF has increased game playing in the research community. For example, activities such as specific staff recruitment or the embellishment of impact.
Most researchers believe that the REF has decreased the authenticity (55 per cent) and novelty (51 per cent) of research in the research community.
The majority (at least 53 per cent or more) of researchers stated that, at an individual level, the REF has not significantly influenced their own research.

Changes to the REF for 2021 and preparations for assessment, for example: eligibility criteria for submissions, open access research practices and approach to impact as measured by the REF

Most academics perceive that changes to the rules for REF 2021 are positive for them.
Early career researchers feel, more than more-established researchers, that changes to the REF affect the expectations, both positive and negative, placed on them.

General attitudes towards the REF

Just under 70 per cent of researchers perceive that, overall, the REF has a negative influence on UK researchers. The big driver of negative attitudes is the burden of the exercise, and specifically that the burden is perceived to outweigh the benefits.
A minority of academics and institutional leaders raised several positive aspects of the REF. These included impacts on individuals for example, the REF values different types of research and the REF has led to positive promotion and hiring decisions. These also included impacts on the HEI sector in the UK. For example, the REF has contributed to a drive on quality over quantity and has facilitated increased investment and funding.

The future of the REF

There is a lack of clarity on the purpose of the REF, and there are a range of views on what the purpose of the REF should be.
Overall, researchers preferred that the top three purposes of the REF should be: accountability, providing an evidence base for priorities, and benchmarking information.
Most survey respondents stated that the REF should not have the purposes of: creating a performance incentive for individuals and HEIs, guiding the allocation of resources within HEIs or informing the allocation of funding to HEIs for research.

What can be learned?

Views on the influence of the REF on academic research and the research community are mixed. The use of the REF as a policy lever and a driver of change and behaviour is often cited as a benefit of REF. Changes made to REF 2021 are broadly seen as positive and there are a range of opinions on what the future of the REF should be. However, there is a perception that the time, costs and workload of the REF outweighs the benefits. There is also a lack of understanding of what the REF is trying to achieve.

It would be important to set out clearly the purpose and desired outcomes of the REF (and align the approach of the exercise to that), as a better understanding of this purpose might help mitigate the perceptions of burden.

In future rounds of the REF, it would be important to act quickly and provide clarity to the sector as early as possible, to make the rules as simple as possible, and ensure rules and changes are communicated clearly.

There is a need for further research and consultation with the sector on the ways to take the exercise forward — for example, looking at international comparisons to learn from other systems that allocate funding and conduct national assessments, as well as considering the options and associated trade-offs of the alterations listed above.