Over the past 10 years, there have been substantial increases in funding for health and biomedical research by public bodies in a number of countries, including Canada, the UK, and the United States, among others. This has coincided with moves towards improved accountability in public service provision in general, and growing demand for both ex ante and ex post evaluations* of research in the health and biomedical research fields in particular. RAND Europe has expertise in both approaches, but has over the past few years undertaken a substantial body of ex ante work on behalf of the English Department of Health using a quantitative methodology known as bibliometrics, which has come increasingly to be used in combination with other evaluation approaches to help inform the Department's decision-making.
What Is Bibliometrics?
Bibliometrics employs quantitative analysis to measure patterns of scientific publication and citation, typically focusing on journal papers. It is one of a set of evaluation methodologies—including case study analysis, peer review, economic rate-of-return analyses and surveys and consultations (among others)—that may be used to help assess the impact of research in the health sciences.
Why Use Bibliometrics?
Bibliometric approaches offer important advantages over other research evaluation methods. They can be used to generate useful quantitative indicators of collaboration and measures of interdisciplinary research. As the sophistication of analytical tools improves, they are being used to develop more general indicators of “quality” and even “excellence.” These analyses are supported by a range of indicators of varying complexity which have been developed over recent years. Robust bibliometric analysis requires a clear understanding of the strengths and limitations of each of these measures, and sensitivity to the contexts in which they are used.
Ultimately, the analytical power of bibliometrics derives mainly from use in combination with other methods, rather than independently. For example, there is growing consensus that bibliometric analysis can be used successfully to complement peer-review decisions. This is particularly the case for large-scale peer-review exercises, where the volume of material to be analysed, and indeed its complexity, may be such that some form of quantitative validation may be useful.
What Does Ex Ante Bibliometric Analysis Look Like in Practice?
RAND Europe has undertaken a substantial body of bibliometric work for the English Department of Health over the past few years, in partnership with the Centre for Science and Technology Studies (CWTS) at the University of Leiden, the Netherlands.** Specifically, bibliometric analysis has been used to support the selection of appropriate academic institutions in the UK as biomedical research centres, academic research departments as biomedical research units, and individuals as faculty members at the National Institute for Health Research. In all three cases, bibliometric analysis was used to evaluate the prior academic performance of applicants, with a view to identifying high achievers and assisting in the selection process.
How Robust Is Bibliometric Analysis?
There are considerable advantages to a bibliometric approach, especially as the power and range of indicators available improves—however, a clear understanding of limitations and caveats is required. From a theoretical perspective, some doubts remain as to the ability of bibliometric methods to capture abstract concepts such as research “quality.” Methodological challenges include issues of journal coverage in major bibliometric databases, adequately identifying author affiliations and choosing the right timeframe for analysis. Caveats to bibliometric analyses include variations in citation behaviour between fields and individuals, and a perennial difficulty in evaluation: attribution. While it is usually possible to determine whether research work contributed to the content of particular publications, often attributing publications solely to particular bodies of research is very difficult, if not impossible.
How Could the Use of Ex Ante Bibliometric Analysis Be Improved?
Further development in some discrete areas could strengthen significantly the analytical power of ex ante bibliometric assessments. First, by investigating the linkages between publication and citation patterns and broader impacts of individual researchers, groups or institutions (such as the health gains resulting from research work, or economic benefits accruing to the wider economy), it may be possible to inform strategic funding decisions to maximise economic returns. Second, new and robust indicators may be developed to identify up-and-coming researchers. Third, cross-checking systems may help to identify the small number of researchers, groups and institutions who effectively “play the system” by manipulating their publication profile—and thus their bibliometric indicators—without any commensurate change in the quality of their research. Other work may include developing indicators of close-to-patient work for funders looking to maximise the impact of their funding on health outcomes.
* There are, broadly speaking, two types of evaluation approach. On the one hand, an ex ante approach seeks to evaluate research proposals before they are conducted: i.e. with a view to selecting the most appropriate proposals for funding support. This is the form of evaluation involved in grant peer-review processes. On the other hand, ex post evaluation involves assessing the impact of research once it has been completed. Ex ante evaluation typically involves some element of ex post analysis, since the prior record of researchers is often reviewed as part of the assessment of whether or not to provide them with further funding support.
** See http://www.cwts.nl/cwtsbv/index.html (accessed 7 August 2009).