The Stern Review was commissioned by the UK government to provide an independent review of the Research Excellence Framework (REF), a method of assessing research from the UK's higher education institutions. The review's aim was threefold: to ensure that the REF was fit-for-purpose, effective and whether there were ways to simplify and reduce the administrative burden; to draw on evidence from the evaluation of REF 2014 and consider other models of research assessment; and to provide options for future iterations of the REF, which would focus on a simpler, light-touch method of research assessment.
The Stern Review's release at the end of July raised two pertinent questions about the REF. What purposes does the REF serve? And does it offer good value for money for the UK's higher education institutions?
There is a wide body of work that discusses the purpose of the REF, and its predecessor, the Research Assessment Exercise (RAE). The RAE was conducted on a (near) quinquennial basis between 1986 and 2013 and assessed research excellence in universities by considering the quality of research publications (known as 'outputs'), and other measures of the research environment, such as number of postgraduate research students, research income and evidence of esteem. The 2014 REF also assessed research excellence, but based it on three main criteria: quality of research outputs, the wider impact of research and the vitality of the research environment.
One key purpose of the REF is to inform the allocation of research funding to UK universities in the amount of nearly £2 billion per year. However, the review is also instrumental in a range of other purposes, including providing an evidence base to inform strategic decisions about national priorities across the research base, a mechanism for accountability, a performance incentive, and a reputational benchmark.
The Stern Review places significant emphasis on concerns about the burden of the process overall...on higher education institutions.
One of the key differences between the 2014 REF and previous RAEs was the introduction of the 'impact' element, meaning that the UK's higher education institutions had to demonstrate the wider impacts and benefits of their research. This is especially important when the research investment comes from public money, making higher education institutions more accountable to the UK taxpayer on the basis of research impacts and not just published research. However, introducing the impact element to the REF has been criticised for making the cost of the assessment exercise too high. In fact, the Stern Review places significant emphasis on concerns about the burden of the process overall, including the impact element, on higher education institutions.
Even if focused solely on the funding allocation that the UK's higher education institutions receive, our cost estimates suggest that the burden of the REF process may not be disproportionate. These show a total cost of £246 million for the REF process, which combines RAND Europe's own estimates of £55 million to cover the impact component of the REF, with £191 million for the other elements. Compared to the likely annual allocation of UK research funding overall of £10 billion—based on the estimate in our report “Assessing impact submissions for REF2014: An evaluation”)—the costs of the process are only around 2.5 percent of the funding allocated.
This percentage is similar to the administrative costs for the UK's research councils for 2015-16, which are around 1.5 to 4.5 percent of their total costs. Unlike our estimates for the REF, these figures do not take into account the costs of the submission process for universities, which—based on our analysis—are likely to make these costs even higher. In other words, our analysis shows that the transaction costs of assessing impact as part of the REF are less than the previously estimated costs of traditional peer reviews.
When making any assessment of costs and burdens, it is also important to look at benefits, both direct and indirect. Our work evaluating the impact element of the REF process (published in 2015 prior to the review) found that the REF provides the opportunity for academics and research users to build better links, and to improve research networks and communication between different groups of research stakeholders. Another RAND Europe study conducted in 2014 found that the REF was effective at deepening the understanding and strategic thinking around impact at the UK's higher education institutions. The REF also allowed academics in the UK to reaffirm their links and relationships with ‘research users', or those in society who will benefit from the breadth and depth of research that occurs in UK universities.
While efficiency should be a goal in research funding, it is not clear whether the REF process is excessively burdensome. In fact, when taking into account the wider benefits of the REF, the process starts to look like good value for money.
Allocating research funding in a fair and balanced way based on evidence takes time, effort and a set of methodologies that match the purpose of the exercise. There are multiple ways to do this, but it is always important to situate the assessment in the wider national context. This is why it is important to ensure that the REF retains and develops the rigour and relevance of the data it collects to allow it to be used for the wide range of purposes.
Based on our analysis of the REF 2014, there are a number of points to consider that could help make the process more rigorous and effective. They include how to manage variations in the way the process is conducted, how to avoid the risk of unsubstantiated and false claims being made, and how to clarify the processes for assessing different kinds of research impact.
The UK has become an international leader in thinking about how to evaluate research and demonstrate its numerous benefits for society. It should continue to focus on these strengths and take pride in the successes of its research and innovation base, using the REF as the empirical evidence to support it.
Molly Morgan Jones is an associate research group director and Susan Guthrie is a senior analyst at RAND Europe.
Commentary gives RAND researchers a platform to convey insights based on their professional expertise and often on their peer-reviewed research and analysis.
