How Do NIHR Peer Review Panels Use Bibliometric Information to Support Their Decisions?

Published in: Scientometrics (2017). doi:10.1007/s11192-017-2417-8

Posted on RAND.org on June 27, 2017

by Salil Gunashekar, Steven Wooding, Susan Guthrie

Read More

Access further information on this document at Springer Netherlands

This article was published outside of RAND. The full text of the article can be found at the link above.

Bibliometrics is widely used as an evaluation tool to assist prospective R&D decision-making. In the UK, for example, the National Institute for Health Research (NIHR) has employed bibliometric analysis alongside wider information in several awarding panels for major funding schemes. In this paper, we examine various aspects of the use of bibliometric information by members of these award selection panels, based on interviews with ten panel members from three NIHR panels, alongside analysis of the information provided to those panels. The aim of the work is to determine what influence bibliometrics has on their decision-making, to see which types of bibliometric measures they find more and less useful, and to identify the challenges they have when using these data. We find that panel members broadly support the use of bibliometrics in panel decision-making, and that the data are primarily used in the initial individual assessment of candidates, playing a smaller role in the selection panel meeting. Panel members felt that the most useful measures of performance are normalised citation scores and the number or proportion of papers in the most highly cited X% (e.g. 5, 10%) for the field. Panel members expressed concerns around the comparability of bibliometrics between fields, but the discussion suggested this largely represents a lack of understanding of bibliometric techniques, confirming that effective background information is important. Based on the evidence around panel behaviour and concerns, we set out guidance around providing bibliometrics to research funding panels.

Research conducted by

This report is part of the RAND Corporation external publication series. Many RAND studies are published in peer-reviewed scholarly journals, as chapters in commercial books, or as documents published by other organizations.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.