Funding and Performance on Clinical Guidelines

Objectives

This article details an exploratory investigation into the characteristics of research cited in two clinical guidelines produced by the National Institute of Health and Clinical Excellence (NICE) in the United Kingdom (UK): that on Dementia, published in 2006, and on Chronic Obstructive Pulmonary Disorder (COPD), published in 2004. More precisely, this article addresses the following questions.

• What are the characteristics of publications cited in these clinical guidelines?
• On those papers with a UK affiliation, who funded the research cited in these clinical guidelines?

The work is part of an overall drive among funders to understand better how research reaches policy and practice.

Method

In order to address these questions, we conducted bibliometric analysis on all the publications cited in the two guidelines and indexed in the bibliometric database the Web of Science. Different bibliometric indicators were selected to assess the various actors' contribution in the research cited in the two guidelines, the collaboration between them in this research and the wider impact of their scientific contribution in the research community.

We subsequently examined the funding acknowledgments in all publications with at least one UK-based author cited in the guidelines under investigation. This was done using information extracted from various bibliographic databases such as the Research Output Databases (ROD) (Dawson et al., 1998; Webster et al., 2003), as well as the full text of these publications.

In addition, this analysis was intended to offer insight into the issue of the time lag between research, its publication in the peer-reviewed literature and its appearance in a clinical guideline—and hence the potential impact of research on policy.

Data Preparation

We began with 491 references for COPD and 775 for Dementia. Extracting and cleaning the references included in each guideline gave us a total of 412 and 616 publications cited in the COPD and the Dementia guidelines respectively.

These publications were cross-referenced against the Web of Science to conduct the bibliometric analysis. Of 412 publications extracted from the COPD guideline, 335 (81.3%) were indexed in the Web of Science. For the publications cited in the Dementia guideline, 494 (80.1%) of the 616 extracted were indexed in the same database.

Of the publications extracted from these two guidelines, we identified those with at least one UK author in order to carry out the funding analysis. Using information from various databases, we found that 148 (35.9%) publications extracted from the COPD had at least one UK author. Of the 616 publications extracted from the Dementia guideline, 228 (37.2%) had at least one UK author.

Main Findings and Future Research Directions

The main findings of our exploratory study are as follows.

Research Funding

• About half of the publications cited in the two guidelines with at least one UK author (40.5% for COPD, 45.6% for Dementia) do not include any funding acknowledgement. The dearth in funding acknowledgement information seems to be related to the age of the publication: more recent publications have more complete funding acknowledgements than older ones.
• Of those publications that did acknowledge funding, the sources included industry, not-for-profit, hospital trusts, government agencies, local or regional authorities, foundations and government departments. Industry was the dominant funder acknowledged by COPD publications, but there was no dominant funder among the Dementia publications.
• Several key UK funders (Medical Research Council [MRC], Department of Health [England], the National Health Service [NHS] and the Wellcome Trust) were acknowledged only to a limited extent, and more frequently in publications cited in the Dementia guideline than in those cited in the COPD guideline.

Characteristics of Publications Cited in Clinical Guidelines

• More than half of the publications cited in both guidelines were published within five years of the guidelines' release, which corroborates the findings from earlier studies on UK clinical guidelines. In their study on UK clinical guidelines related to cancer published up to 2006, Lewison and Sullivan (2008) found a peak of cited publications after 2000. Other works on UK clinical guidelines discovered that a significant share of the publications they cite are published within 10 years prior to the release of these guidelines (for example, Grant et al., 2000; Buxton et al., 2009).
• More than 35% of publications cited in these two UK clinical guidelines featured at least one author with a UK affiliation. This proportion is high, given that the UK share in all publications published in medical research at the global level was approximately 8.6% in 2006 (Hassan, 2009).* This “overrepresentation” of UK-authored publications in UK clinical guidelines is in line with the results of previous studies, although earlier studies have found overall lower shares of UK-authored publications cited in UK clinical guidelines (Grant et al., 2000; Lewison and Sullivan, 2008; Buxton et al., 2009).
• The publications cited in the two guidelines have a high impact in the scientific community, as reflected in the field-normalised citation score. Publicly-funded research institutes produced the papers with the highest citation scores (more than 8 for COPD and around 24 for Dementia), with companies ranking second in terms of citation scores. Similarly, previous studies on UK clinical guidelines have underlined the high scientific impact of the publications they cite.
• Most of the publications (more than 80%) cited in both guidelines are copublications between authors from different institutions at the national or international level.

The results of this exploratory work raise several questions for future research.

Specific Questions Related to Clinical Guidelines

• Is there a systematic selection bias, such that the authors of the UK clinical guidelines are more likely to cite UK-authored and UK-focused research? Obviously, this would be expected if they concern local health issues, but could the bias extend beyond this?
• Is there a bias towards the inclusion of highly-cited publications in UK clinical guidelines? Does inclusion of a research paper in a clinical guideline lead to subsequent citations?
• What type of research (e.g. basic, clinical) is cited most often in UK clinical guidelines?
• Can the publications cited in clinical guidelines be used to help us estimate the time lags between research, publication and potential influence on policy and practice resulting from inclusion in a clinical guideline?

Broader Questions Related to the Relation Between Research Funding and Performance

• What is the relationship between the institutional origin of research funding and scientific performance?
• What is the relationship between the type of research funding and scientific performance?

References

Buxton, M., S. Hanney, S. Morris, L. Sundmacher, J. Mestre-Ferrandiz, M. Garau, J. Sussex, J. Grant, S. Ismail, E. Nason, S. Wooding, and S. Kapur, Medical Research: What's It Worth? Estimating the Economic Benefits from Medical Research in the UK, London: Health Economics Research Group, Office of Health Economics/RAND Europe, EP-200800-10, 2009.

Dawson, G., B. Lucocq, R. Cottrell, and G. Lewison, Mapping the Landscape: National Biomedical Research Outputs 1988–95, London: Wellcome Trust, 1998. As of April 2010:
http://www.wellcome.ac.uk/stellent/groups/corporatesite/@policy_communications/documents/web_document/wtd003187.pdf

Esterle, L., B. Amiranoff, and A. Baneyx, “La Recherche Publique dans le Domaine Biomédical en France: Analyse Quantitative et Éléments de Diagnostic,” Etude Bioméd, May 2008. As of April 2010:
http://media.education.gouv.fr/file/2008/54/6/RapportBiomed-VF_31546.pdf

Grant, J., R. Cottrell, F. Cluzeau, and G. Fawcett, “Evaluating 'Payback' on Biomedical Research from Papers Cited in Clinical Guidelines: Applied Bibliometric Study,” British Medical Journal, Vol. 320, No. 7242, 2000, pp. 1107–1111.

Hassan, E., Health and Medical Research in France: Observatory on Health Research Systems, Cambridge, United Kingdom: RAND Europe, DB-590-DH, 2009. As of May 6, 2012:
http://www.rand.org/pubs/documented_briefings/DB590.html

Lewison, G., and R. Sullivan, “The Impact of Cancer Research: How Publications Influence UK Cancer Clinical Guidelines,” British Journal of Cancer, Vol. 98, No. 12, 2008, pp. 1944–1950.

Webster, B., G. Lewison, and I. Rowlands, Mapping the Landscape II: Biomedical Research in the UK, 1989–2002. London: Department of Information Science, City University, 2003. As of April 2010:
http://www.homepages.ucl.ac.uk/~uczciro/mtl2chapter%201.pdf

Note

* The UK share in biomedical sciences at the global level was calculated by the French Observatoire des Sciences et des Techniques using the Web of Science database (Esterle et al., 2008). This share was calculated based on the fractional counting of multi-authored publications, while in this document we use whole counting. Thus, our results might overestimate slightly the share of cited publications featuring at least one author with a UK affiliation.