Evaluating Grant Peer Review in the Health Sciences: A review of the literature
Aug 11, 2009
In 2009, RAND Europe conducted a literature review in order to assess the effectiveness and efficiency of peer review for grant funding. This report presents an update to that review to reflect new literature on the topic, and adds case studies exploring peer review practice at six international funders.
An updated review of the literature and six case studies
|PDF file||3.3 MB||
Use Adobe Acrobat Reader version 10 or higher for the best experience.
Though often viewed as the 'gold standard' process of quality assurance for research, grant peer review has also received significant criticism from both within and outside academia. Detractors highlight inefficiency and structural flaws that compromise its effectiveness in allocating funding. In 2009, RAND Europe conducted a review of the literature to evaluate these criticisms. This new report updates that literature review, and also provides case studies of current practice across six major international biomedical and health research funders. The work was commissioned by the Canadian Institutes of Health Research (CIHR) with the aim of supporting the ongoing review of their peer review system, particularly the forthcoming work of the Peer Review Expert Panel convened to review the design and adjudication processes of CIHR's investigator-initiated research programmes.
Overall, the evidence is this. Judging whether peer review is demonstrably better than any other system is impossible because of the lack of comparators. No funding agencies have made significant use of other allocation systems. Even comparisons between or research on peer review systems is limited, with most studies examining the peer review process of one particular funder in one particular context, and few go beyond process measures to judge improvement. Considering the scope and costs of peer review internationally, this gap should be addressed.
The evidence that is available suggests the burden of peer review largely falls on applicants (rather than reviewers), but this typically receives less attention since it is less visible to funders. Additionally, application rates are increasing, so there is a need to reduce applicant burden, perhaps though reducing the complexity of the application process (though careful evaluation would be needed to ensure this is effective). In terms of effectiveness, one key challenge is determining how 'good' is 'good enough' in terms of reliability, discernment or bias. Clearly there is uncertainty in peer review decisions, but most funders do not use or even acknowledge this in their assessment processes. There is clear evidence that peer review can stifle innovation, and that it is not a good predictor of future (bibliometric) performance, which may be related to the broader pool of evidence that peer review ratings are inconsistent. There is more limited evidence of bias in peer review in relation to cognitive familiarity and due to cronyism.
Grant peer review: the context
Systems of grant peer review: international practice
Evaluating grant peer review: what do we know?
Case study template and interview protocol
The research described in this report was funded by the Canadian Institutes of Health Research and conducted by RAND Europe.
This report is part of the RAND Corporation Research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.
This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.
The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.