What do we know about grant peer review in the health sciences?

An updated review of the literature and six case studies

by Susan Guthrie, Ioana Ghiga, Steven Wooding

Download eBook for Free

FormatFile SizeNotes
PDF file 3.3 MB

Use Adobe Acrobat Reader version 10 or higher for the best experience.

Research Questions

  1. How effective is peer review for grant funding in the biomedical and health sciences?
  2. What is the burden of peer review in this context?
  3. What is accepted practice in terms of peer review for grant funding at major biomedical and health science research funders internationally?

Though often viewed as the 'gold standard' process of quality assurance for research, grant peer review has also received significant criticism from both within and outside academia. Detractors highlight inefficiency and structural flaws that compromise its effectiveness in allocating funding. In 2009, RAND Europe conducted a review of the literature to evaluate these criticisms. This new report updates that literature review, and also provides case studies of current practice across six major international biomedical and health research funders. The work was commissioned by the Canadian Institutes of Health Research (CIHR) with the aim of supporting the ongoing review of their peer review system, particularly the forthcoming work of the Peer Review Expert Panel convened to review the design and adjudication processes of CIHR's investigator-initiated research programmes.

Key Findings

Overall, the evidence is this. Judging whether peer review is demonstrably better than any other system is impossible because of the lack of comparators. No funding agencies have made significant use of other allocation systems. Even comparisons between or research on peer review systems is limited, with most studies examining the peer review process of one particular funder in one particular context, and few go beyond process measures to judge improvement. Considering the scope and costs of peer review internationally, this gap should be addressed.

The evidence that is available suggests the burden of peer review largely falls on applicants (rather than reviewers), but this typically receives less attention since it is less visible to funders. Additionally, application rates are increasing, so there is a need to reduce applicant burden, perhaps though reducing the complexity of the application process (though careful evaluation would be needed to ensure this is effective). In terms of effectiveness, one key challenge is determining how 'good' is 'good enough' in terms of reliability, discernment or bias. Clearly there is uncertainty in peer review decisions, but most funders do not use or even acknowledge this in their assessment processes. There is clear evidence that peer review can stifle innovation, and that it is not a good predictor of future (bibliometric) performance, which may be related to the broader pool of evidence that peer review ratings are inconsistent. There is more limited evidence of bias in peer review in relation to cognitive familiarity and due to cronyism.

Recommendations

  • Given the centrality of the peer review process to driving the allocation of resources in the current system of science, there is a need for better evidence, not only on its overall effectiveness but also to support the design of improved peer review processes.
  • Funders should be more willing to experiment with, evaluate and publish results from evaluations of alternative approaches.
  • It is clear from the evidence that applicant burden needs to be considered alongside reviewer and administrative burden. But it must be borne in mind that simplifying application processes may not reduce the effort expended on them.

Table of Contents

  • Chapter One

    Introduction

  • Chapter Two

    Methodology

  • Chapter Three

    Grant peer review: the context

  • Chapter Four

    Systems of grant peer review: international practice

  • Chapter Five

    Evaluating grant peer review: what do we know?

  • Chapter Six

    Conclusions

  • Appendix A

    Case study template and interview protocol

  • Appendix B

    Case studies

Research conducted by

The research described in this report was funded by the Canadian Institutes of Health Research and conducted by RAND Europe.

This report is part of the RAND Corporation research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.