Comparative Scientometric Assessment of the Outputs of ERC-funded Projects

Background

Founded in 2007, the European Research Council (ERC) is a public body that funds ‘investigator-driven’ research conducted within the European Union (EU). The ERC is a key element of the European Commission’s Horizon 2020 programme, the EU’s biggest ever research and innovation programme, with nearly €80 billion of funding available over seven years (2014 to 2020). The main goal of the ERC is to encourage high quality research in Europe through competitive funding that is awarded on the basis of scientific excellence.

Goals

The European Research Council Executive Agency (ERCEA) asked RAND Europe and Observatoire des sciences et des technologies (OST) to carry out a comprehensive scientometric assessment of the outputs of ERC grants – the first time that a full analysis of the ERC’s funded research had been conducted in this way. The evaluation took a comparative perspective, examining ERC-funded publications in relation to those of other US and EU funding agencies, and delivered a robust framework which will allow for continuous updates of ERC scientific achievements in the future. The results were documented through a series of reports.

Methodology

The study used four distinct approaches to assess the outputs stemming from ERC grants, three of which were quantitative assessments of the entire corpus of ERC-funded work (incorporating bibliometrics, patent analysis, and altmetrics), and one which involved a qualitative analysis of a sample of the most highly cited papers. All analyses were carried out across the three domains of research supported by the ERC: Life Sciences, Physical Sciences and Engineering, and Social Sciences and Humanities.

Key Findings

In relation to the four research questions, the study produced the following findings.

1. Is the ERC peer review process successful in selecting the best candidates among those who have submitted a proposal?

• The bibliometric analysis showed that ERC competitions attract high-profile researchers and the assessment of its peer review committees tend to select those who have published numerous high impact scientific papers (in comparison to those not funded).
• There was little difference in interdisicplinarity between funded and non-funded researchers, the analysis showing that this was largely determined by the domain of research (interdisciplinarity was highest in the Life Sciences).
• The patent analysis found that, overall, ERC-funded researchers tended to have hgher inventive activity at the time of funding than those not funded (although there were some exceptions among Life Sciences panels).

2. Does the funding provided by the ERC help grantees improve their scientific output and impact?

• The bibliometric analysis did not find evidence of a major impact on the publications output of funded researchers, whether in terms of productivity (where there is a slight increase), scientific impact or interdisciplinarity.
• While data limitations prevent definitive conclusions being drawn from the patent analysis, the pattern tended to indicate that ERC funding had a slight positive effect for junior researchers but did not affect the productivity of senior researchers.
• It was not possible to measure a change over time in altmetric scores, as data were not available at the point funding was awarded. However, funded researchers obtained higher altmetric scores than unfunded, albeit with differences between fields: natural and physical sciences were picked up on social media at a much lower rate than medical disciplines or social sciences and humanities.

3. Do ERC grantees perform better than researchers funded by other European and American funding agencies?

• The bibliometric analysis show that ERC-funded researchers’ publication output is consistently among the best of the funding agencies considered in the study, both in terms of productivity and scientific impact.
• When looking at patents, ERC-funded researchers showed lower inventive activity than researchers funded by US agencies, but they filed for more patents from a single invention (possibly to due to geographical and/or economic reasons).
• The altmetric analysis found that ERC-funded researchers generally obtained higher Mendeley scores than those funded by the other agencies studied. However, for other altmetric indicators – and in particular, Twitter – researchers supported by US funders typically scored more highly.

4. What kinds of contributions to science and knowledge are made by the most highly cited ERC research?

• In the qualitative assessment, reviewers considered 21% of the papers reviewed (all of which were selected from the top 1% most cited) to have made a ‘landmark contribution’ to their respective fields.
• A further 61% of the ‘top’ papers were considered to have made a ’significant contribution to science’ or ‘major addition to knowledge’.
• Considering impact beyond academia, 23% of papers were expected to lead to benefits in their sector of practice, but economic, social and cultural impacts were anticipated rarely.
• It should be noted that this part of the study did not look at all ERC-funded research, instead considering a small sample of publications (95 reviews of 56 ‘top’ papers). As such the analysis cannot be considered representative of the wider body of ERC-supported work.

---

Read the full study