Because of the growing demand for transparency and accountability in research evaluation, researchers developed a comprehensive list of evaluation tools and techniques and explained when each might be most useful and why. They then offered a list of 100 ideas for metrics that can be used to assess and communicate the value of biomedical research.
Background
There is growing demand internationally for research evaluation, due to an increasing emphasis on governance and accountability in both the public and private sectors.
The traditional metrics used internally by research institutions (the size of a grant, the number of grants, number of academic publications, etc.) can serve as surrogate measures of research quality given that they are based on peer review. However, such traditional evaluations are not effective with the external audiences that are important for policymaking, which are increasingly focused on the wider health, social and economic impacts of the work being done.
Goals
In response to increasing expectations of accountability and transparency, the Association of American Medical Colleges (AAMC), in collaboration with RAND Europe, undertook a project to help communicate the wider value of biomedical research. The initiative developed resources to support academic medical centres in evaluating the outcomes and impacts of their research using approaches relevant to various stakeholders, including patients, providers, administrators, and legislators. Two main publications have resulted from this project, which together seek to support the development of evaluation frameworks.
In the first report, Measuring research: A guide to research evaluation frameworks and tools, prepared for the Association of American Medical Colleges, we have compiled a comprehensive list of evaluation tools and techniques. We discuss when each might be most useful; examine the advantages and disadvantages of different approaches; and look at the context in which each of 14 different frameworks has been used before. We also present a decision tree for funders, policymakers and researchers who want to evaluate research and need practical guidance on how to choose the appropriate approach.
The second report presents 100 ideas for metrics that can be used assess and communicate the value of biomedical research. The list is not comprehensive, and the metrics are not fully developed, but they should serve to stimulate and broaden thinking about how academic medical centres can communicate the value of their research to a broad range of stakeholders. Some metrics, such publication numbers, are measurable in the short and medium terms. Others, such as community-level health and economic outcomes, are longer-term and appeal to audiences other than academic medical centres.
Findings
Findings from the first report include:
Designing a research evaluation framework requires trade-offs: there is no silver bullet.
To be effective, the design of the framework should depend on the purpose of the evaluation: advocacy, accountability, analysis and/or allocation.
Research evaluation tools typically fall into one of two groups, which serve different needs; multiple methods are required if researchers’ needs span both groups.
The units of aggregation used for collecting, analysing and reporting data will depend on the target audience(s), and the tool(s) used.
There are some perennial challenges to research evaluation that need to be addressed.
Research evaluation approaches need to suit their wider context.
Finally, implementation needs ownership, the right incentives and support.
Findings from the second report include:
Different types of metrics – Nine types of metrics were identified:
Research – the direct development and outputs arising from research
Prestige – the external recognition of the research
Career – the teaching and career development of researchers following the research
Process – the effectiveness and efficiency of the administrative and institutional processes supporting the research
Network —the interactions of researchers and academic institutions with external stakeholders
Policy — the changes in policies to which the research has contributed
Health — the changes in health outcomes to which the research has contributed
Economics — the changes in the (local) economy to which the research has contributed
Broader — changes across the range of categories listed above
Long term vs short term – Some metrics, such as publication numbers, are measurable in the short and medium terms. Others, such as community-level health and economic outcomes, are longer-term and appeal to audiences other than academic medical centres.
Stakeholders – Each metric is likely to be of interest to one or more stakeholder groups, which include funders and donors, external academic stakeholders, board management, local population, research management, and patients.
Sources – Each metric draws on data collected through one or more sources.
