Aug 11, 2009
In 2009, RAND Europe conducted a literature review in order to assess the effectiveness and efficiency of peer review for grant funding. This report presents an update to that review to reflect new literature on the topic, and adds case studies exploring peer review practice at six international funders.
Though often viewed as the 'gold standard' process of quality assurance for research, grant peer review has also received significant criticism from both within and outside academia. Detractors highlight inefficiency and structural flaws that compromise its effectiveness in allocating funding. In 2009, RAND Europe conducted a review of the literature to evaluate these criticisms. This new report updates that literature review, and also provides case studies of current practice across six major international biomedical and health research funders. The work was commissioned by the Canadian Institutes of Health Research (CIHR) with the aim of supporting the ongoing review of their peer review system, particularly the forthcoming work of the Peer Review Expert Panel convened to review the design and adjudication processes of CIHR's investigator-initiated research programmes.
Overall, the evidence is this. Judging whether peer review is demonstrably better than any other system is impossible because of the lack of comparators. No funding agencies have made significant use of other allocation systems. Even comparisons between or research on peer review systems is limited, with most studies examining the peer review process of one particular funder in one particular context, and few go beyond process measures to judge improvement. Considering the scope and costs of peer review internationally, this gap should be addressed.
The evidence that is available suggests the burden of peer review largely falls on applicants (rather than reviewers), but this typically receives less attention since it is less visible to funders. Additionally, application rates are increasing, so there is a need to reduce applicant burden, perhaps though reducing the complexity of the application process (though careful evaluation would be needed to ensure this is effective). In terms of effectiveness, one key challenge is determining how 'good' is 'good enough' in terms of reliability, discernment or bias. Clearly there is uncertainty in peer review decisions, but most funders do not use or even acknowledge this in their assessment processes. There is clear evidence that peer review can stifle innovation, and that it is not a good predictor of future (bibliometric) performance, which may be related to the broader pool of evidence that peer review ratings are inconsistent. There is more limited evidence of bias in peer review in relation to cognitive familiarity and due to cronyism.
Grant peer review: the context
Systems of grant peer review: international practice
Evaluating grant peer review: what do we know?
Case study template and interview protocol