Standardising quantitative indicators of impact within REF case studies
The Research Excellence Framework (REF) 2014 was a nationwide exercise to assess the quality of research in UK higher education institutions. The next REF will be carried out in 2021.
To inform the development of the guidance and criteria for the preparation of impact case studies in REF 2021, we examined the case studies submitted to REF 2014 and identified quantitative indicators of impact. We then developed guidance for how these indicators could be standardised for potential use in REF 2021.
The Research Excellence Framework (REF), first conducted in 2014, assesses the quality of research in UK higher education institutions (HEIs). REF 2014 demonstrated that research conducted by universities can lead to impact beyond academia – for example, on society, the economy, public policy, and health – within the UK and beyond. In REF 2014, HEIs were invited to submit examples of impact as case studies. Combining a compelling narrative, with the underpinning research and citations of evidence to corroborate the impact, was seen as a 'workable approach'. Accordingly, in REF 2021, impact will continue to be assessed through the use of case studies.
RAND Europe was commissioned by HEFCE (now Research England) on behalf of the four UK higher education funding bodies, to:
- Identify quantitative indicators of impact in the case studies submitted to REF 2014; and
- Develop guidance for how these indicators could be standardised for potential use in REF 2021.
The study is intended to inform the development of the guidance and criteria for the preparation of impact case studies in REF 2021.
We adopted a mixed-methods approach to address the study objectives. This involved a blend of text mining and qualitative techniques:
- Rapid review of the literature to understand the kinds of quantitative indicators that might be used in impact case studies (to inform our search of the REF 2014 case studies)
- Construction and analysis of the data set of sentences containing quantitative indicators of impact from the REF 2014 impact case studies (using text mining and qualitative techniques)
- Initial workshop to identify areas for potential standardisation and develop the corresponding standards themselves
- Testing of proposed guidance (involving consultation with the Forum for Responsible Research Metrics and interviews with individuals working within the higher education sector)
- Synthesis of evidence and reporting
We propose guidance for two broad categories of standards. First, we outline a ‘style guide’ that contains general stylistic items that could be standardised in order to make quantitative indicators of impact, and specific formulations of them, more discoverable in the case studies in general. Second, we detail ‘specific guidance’ that addresses more specific and commonly occurring quantitative indicators that have been used as evidence of impact in the REF 2014 case studies.
This consists of general stylistic items that can be standardised to make quantitative indicators of impact, and specific formulations of them, more discoverable in the case studies. The style guide would apply across all of the specific guidance and include:
- Percentages and rates
- Measures of change
- Time periods
This covers more specific and commonly occurring quantitative indicators that have been used as evidence of impact in the case studies. Standardisation could improve discoverability of quantitative indicators of impact and their potential aggregation. This includes:
- Mentions in non-academic documents and the media
- Financial figures