In 2009, RAND Europe conducted a literature review in order to assess the effectiveness and efficiency of peer review for grant funding. This report presents an update to that review to reflect new literature on the topic, and adds case studies exploring peer review practice at six international funders.

What do we know about grant peer review in the health sciences?

An updated review of the literature and six case studies

Susan Guthrie, Ioana Ghiga, Steven Wooding

ResearchPublished Jun 7, 2018

Though often viewed as the 'gold standard' process of quality assurance for research, grant peer review has also received significant criticism from both within and outside academia. Detractors highlight inefficiency and structural flaws that compromise its effectiveness in allocating funding. In 2009, RAND Europe conducted a review of the literature to evaluate these criticisms. This new report updates that literature review, and also provides case studies of current practice across six major international biomedical and health research funders. The work was commissioned by the Canadian Institutes of Health Research (CIHR) with the aim of supporting the ongoing review of their peer review system, particularly the forthcoming work of the Peer Review Expert Panel convened to review the design and adjudication processes of CIHR's investigator-initiated research programmes.

Key Findings

Overall, the evidence is this. Judging whether peer review is demonstrably better than any other system is impossible because of the lack of comparators. No funding agencies have made significant use of other allocation systems. Even comparisons between or research on peer review systems is limited, with most studies examining the peer review process of one particular funder in one particular context, and few go beyond process measures to judge improvement. Considering the scope and costs of peer review internationally, this gap should be addressed.

The evidence that is available suggests the burden of peer review largely falls on applicants (rather than reviewers), but this typically receives less attention since it is less visible to funders. Additionally, application rates are increasing, so there is a need to reduce applicant burden, perhaps though reducing the complexity of the application process (though careful evaluation would be needed to ensure this is effective). In terms of effectiveness, one key challenge is determining how 'good' is 'good enough' in terms of reliability, discernment or bias. Clearly there is uncertainty in peer review decisions, but most funders do not use or even acknowledge this in their assessment processes. There is clear evidence that peer review can stifle innovation, and that it is not a good predictor of future (bibliometric) performance, which may be related to the broader pool of evidence that peer review ratings are inconsistent. There is more limited evidence of bias in peer review in relation to cognitive familiarity and due to cronyism.

Recommendations

  • Given the centrality of the peer review process to driving the allocation of resources in the current system of science, there is a need for better evidence, not only on its overall effectiveness but also to support the design of improved peer review processes.
  • Funders should be more willing to experiment with, evaluate and publish results from evaluations of alternative approaches.
  • It is clear from the evidence that applicant burden needs to be considered alongside reviewer and administrative burden. But it must be borne in mind that simplifying application processes may not reduce the effort expended on them.

Topics

Document Details

Citation

RAND Style Manual
Guthrie, Susan, Ioana Ghiga, and Steven Wooding, What do we know about grant peer review in the health sciences? An updated review of the literature and six case studies, RAND Corporation, RR-1822-CIHR, 2018. As of October 13, 2024: https://www.rand.org/pubs/research_reports/RR1822.html
Chicago Manual of Style
Guthrie, Susan, Ioana Ghiga, and Steven Wooding, What do we know about grant peer review in the health sciences? An updated review of the literature and six case studies. Santa Monica, CA: RAND Corporation, 2018. https://www.rand.org/pubs/research_reports/RR1822.html.
BibTeX RIS

Research conducted by

The research described in this report was funded by the Canadian Institutes of Health Research and conducted by RAND Europe.

This publication is part of the RAND research report series. Research reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND research reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.