Biomedical research affects society in many ways. It has been shown to improve health, create jobs, add to our knowledge, and foster new collaborations. Despite the complexity of modern research, many of the metrics used to evaluate the impacts of research still focus on the traditional, often academic, part of the research pathway, covering areas such as the amount of grant funding received and the number of peer-reviewed publications. In response to increasing expectations of accountability and transparency, the Association of American Medical Colleges (AAMC), in collaboration with RAND Europe, undertook a project to help communicate the wider value of biomedical research. The initiative developed resources to support academic medical centers in evaluating the outcomes and impacts of their research using approaches relevant to various stakeholders, including patients, providers, administrators, and legislators. This study presents 100 ideas for metrics that can be used to assess and communicate the value of biomedical research. The list is not comprehensive, and the metrics are not fully developed, but they should serve to stimulate and broaden thinking about how academic medical centers can communicate the value of their research to a broad range of stakeholders.
Biomedical research affects society in many ways. It has been shown to improve health, create jobs, add to our knowledge, and foster new collaborations. Despite the complexity of modern research, many of the metrics used to evaluate the impacts of research still focus on the traditional, often academic, part of the research pathway, covering areas such as the amount of grant funding received and the number of peer-reviewed publications.
In response to increasing expectations of accountability and transparency, the Association of American Medical Colleges (AAMC), in collaboration with RAND Europe, undertook a project to help communicate the wider value of biomedical research. The initiative developed resources to support academic medical centers in evaluating the outcomes and impacts of their research using approaches relevant to various stakeholders, including patients, providers, administrators, and legislators.
This article presents 100 ideas for metrics that can be used assess and communicate the value of biomedical research. The list is not comprehensive, and the metrics are not fully developed, but they should serve to stimulate and broaden thinking about how academic medical centers can communicate the value of their research to a broad range of stakeholders. Some metrics, such publication numbers, are measurable in the short and medium terms. Others, such as community-level health and economic outcomes, are longer-term and appeal to audiences other than academic medical centers.
The metrics included in this study do not represent a systematic set. Rather, the metrics were identified and selected in the course of several steps as part of a larger research project. First, RAND Europe reviewed research evaluation frameworks used internationally, along with the tools/methods these frameworks included.1 Second, workshops were held with medical college faculty and research leaders to identify appropriate approaches for American medical colleges alongside identifying the key stakeholder audiences for the evaluation. In a third phase, three key external stakeholder groups—community members, research administrators, and state legislators—confirmed the salience of the identified metrics.2 Metrics identified through these three steps, combined with the knowledge of the research team of existing metrics, formed the foundation for the current list of metrics.
Not all the metrics and areas will be of interest or applicable to all biomedical research institutions and organizations. This article, rather, provides a long list of the metrics from which a relevant subset could be selected, depending on the specific institutional context.
Metric Classifications
To help identify relevant metrics for a particular application, we have classified the metrics in three ways:
Stakeholders
Each metric is likely to be of interest to one or more stakeholder groups. This section maps each metric against the relevant stakeholders:
"State and local legislators" refers to policymakers. In this study, the focus is on their interests as policymakers and representatives of the region and the local population. Any role that these individuals may have as funders in this context is not considered—those interests are all captured under the "funders and donors" category.
"Funders and donors" includes any external group or individual providing funding for research, researchers, or research equipment. This includes the National Institutes of Health (NIH) as well as smaller funders and donors, both public and private.
"External academic stakeholders" are any academics based at another institution. These academics can be potential collaborators or people that the institution might want to recruit.
"Board management" (of the research institution) refers to the person or group of people responsible for oversight of all the functions of the institution, not just research.
"Community" refers to the local population served by the institution (which will include but is not limited to patients). Of particular interest are those who might be engaged in the research process in some way (community partners, trial participants).
"Research management" (within the research institution) refers to the person or group of people responsible for oversight of the research function at the institution specifically.
"Patients" refers to people who are being treated or have recently been treated at the institution.
Metrics for Assessing and Communicating the Value of Biomedical Research, by Stakeholder Group
Research impacts
Number of journal articles published: Funders and donors, external academic stakeholders, board management, research management
Number of citations: Funders and donors, external academic stakeholders, board management, community, research management
Number of research output downloads: Funders and donors, external academic stakeholders, board management, research management
Mentions in social media: State and local legislators, funders and donors, board management, research management
Number and size of grant awards: State and local legislators, funders and donors, external academic stakeholders, board management, research management
Number and size of awards from major funders: State and local legislators, funders and donors, external academic stakeholders, board management, research management
Number of different research funders supporting research: State and local legislators, funders and donors, external academic stakeholders, board management, research management
Success rate of applications: External academic stakeholders, board management, research management
Catalogue of infrastructure: State and local legislators, funders and donors, external academic stakeholders, board management, research management
Use of infrastructure by other researchers: State and local legislators, funders and donors, external academic stakeholders, board management, research management
Measures of prestige
Number of editorships of high profile journals: Funders and donors, external academic stakeholders, board management, research management
Number of staff on relevant boards and committees: State and local legislators, funders and donors, external academic stakeholders, board management, community, research management, patients
Number of academy members: State and local legislators, funders and donors, external academic stakeholders, board management, research management
Number and type of prizes: State and local legislators, funders and donors, external academic stakeholders, board management, research management
Number of (international) speaker invitations/conference invitations: State and local legislators, funders and donors, external academic stakeholders, board management, research management
Number of media mentions: State and local legislators, funders and donors, external academic stakeholders, board management, research management
Number of applications per open post: External academic stakeholders, board management, research management
Percentage of out-of-state and international applications per research job/PhD post: External academic stakeholders, board management, research management
Track record of new hires: State and local legislators, external academic stakeholders, board management, research management
Undergraduate applications: State and local legislators, external academic stakeholders, board management, community
Grade Point Average of incoming students: Board management
Teaching and career development impacts
Grade Point Average of graduates: State and local legislators, board management
Longitudinal data on career progression of students: State and local legislators, funders and donors, board management, community, research management
Number of PhD graduates: External academic stakeholders, board management, research management
Completion rate of PhD graduates: External academic stakeholders, board management, research management
Number of publications per PhD graduate: External academic stakeholders, board management, research management
5/10/15-year career outcomes for PhD graduates: Funders and donors, external academic stakeholders, board management, research management
K to R conversion rate: External academic stakeholders, board management, research management
Career outcomes for researchers: Funders and donors, external academic stakeholders, board management, research management
Subject coverage of the professional development programme: External academic stakeholders, board management, research management
Uptake of the professional development programme: External academic stakeholders, board management, research management
Feedback on the professional development programme: External academic stakeholders, board management, research management
Improved educational attainment/reduced drop out rate: State and local legislators, funders and donors, board management, community, research management
Research and institutional processes
Start-up time for research projects: State and local legislators, funders and donors, board management, community, research management, patients
Start-up time for clinical trials: State and local legislators, funders and donors, external academic stakeholders, board management, community, research management, patients
Average time from funding to publication: State and local legislators, funders and donors, board management, research management
Proportion of funds spent on administration: State and local legislators, funders and donors, board management, research management
Support staff to researcher ratio: State and local legislators, funders and donors, external academic stakeholders, board management, research management
Prompt payment of community partners: State and local legislators, funders and donors, external academic stakeholders, board management, community, research management
How hiring decisions are made: External academic stakeholders, board management, research management
How decisions are made to apply for grants: Funders and donors, external academic stakeholders, board management, community, research management, patients
How publications decisions are made: External academic stakeholders, board management, community, research management, patients
Proportion of publications that are open access: State and local legislators, funders and donors, external academic stakeholders, board management, community, research management, patients
Proportion of trials where protocol and findings are published: State and local legislators, funders and donors, external academic stakeholders, board management, community, research management, patients
Description of institution's policy on health equity in research: State and local legislators, funders and donors, board management, community, patients
Proportion of projects which consider health equity in their design and conduct: State and local legislators, funders and donors, external academic stakeholders, board management, community, patients
Networks and dissemination
Number of collaborations on grant applications and projects: Funders and donors, external academic stakeholders, community, research management
Levels of co-authorship: Funders and donors, external academic stakeholders, community, research management
Bilbiometric networks: Fxternal academic stakeholders, community, research management
Total number of different collaborators across all projects: Funders and donors, external academic stakeholders, community, research management
Description of range of collaborations: Funders and donors, external academic stakeholders, community, research management
Number of research projects engaging community partners: State and local legislators, funders and donors, external academic stakeholders, board management, community, research management, patients
Number of research projects engaging community partners for the entire duration of the project: State and local legislators, funders and donors, board management, community, research management
Number of articles co-authored with community partner: State and local legislators, funders and donors, community
Existence of specifically tailored material for different community groups: State and local legislators, funders and donors, external academic stakeholders, board management, community, research management, patients
Size of communications office: Funders and donors, board management
Number of staff engaged in outreach: Funders and donors, board management, community, research management
Number of people attending outreach events and their perceptions: State and local legislators, funders and donors, board management, community, research management
Level of participation in clinical trials: State and local legislators, funders and donors, external academic stakeholders, board management, community, research management, patients
Number of projects with an industry partner: State and local legislators, funders and donors, external academic stakeholders, board management, research management
Industrial research funding for PhD/secondment positions in industry and PhD scholarships: State and local legislators, funders and donors, external academic stakeholders, board management, research management
Number of policy secondments: State and local legislators, funders and donors, board management
Policy impacts
Number of invitations from policy makers: State and local legislators, funders and donors, board management
Number of citations on clinical guidelines: State and local legislators, funders and donors, board management, research management, patients
Number of citations in policy documents: State and local legislators, funders and donors, board management, community, research management, patients
Health impacts
Improved health of patients: State and local legislators, funders and donors, board management, community, research management, patients
Improved quality of care metrics: State and local legislators, board management, community, research management, patients
Number of lives touched: State and local legislators, funders and donors, board management, community
Narrowing of health/health-care disparities: State and local legislators, funders and donors, board management, community, patients
Improved awareness of preventative measures in the community: State and local legislators, funders and donors, board management, community, research management, patients
Number of treatment developed in house: State and local legislators, funders and donors, external academic stakeholders, board management, community, research management, patients
Number of new treatments available (adopted from elsewhere): Funders and donors, board management, community, patients
Percentage, number, and range of types of clinicians on research projects: Funders and donors, external academic stakeholders, board management, research management
Number of uses of research infrastructure in clinical practice: State and local legislators, funders and donors, external academic stakeholders, board management, research management, patients
Economic impacts and commercialisation
Level of (local) spending: State and local legislators, board management, community
Amount of direct employment: State and local legislators, board management, community
Number and type of new offices (including subsidiaries) in the area: State and local legislators, funders and donors, external academic stakeholders, board management, community, research management, patients
Size of tech transfer office: State and local legislators, external academic stakeholders, board management, research management
Existence of Intellectual Property policy: State and local legislators, funders and donors, external academic stakeholders, board management, research management
Number of patent applications: State and local legislators, funders and donors, external academic stakeholders, board management, research management
Number of patents awarded: State and local legislators, funders and donors, external academic stakeholders, board management, research management
Number of patent citations: State and local legislators, funders and donors, external academic stakeholders, board management, research management
Number of licensing agreements and licensing revenue: State and local legislators, funders and donors, external academic stakeholders, board management, research management
List/examples of know-how taken up by industry: State and local legislators, funders and donors, external academic stakeholders, board management, research management
Number of private sector innovations/products/devices brought to market: State and local legislators, funders and donors, external academic stakeholders, board management, research management
Number of spin outs: State and local legislators, funders and donors, external academic stakeholders, board management, research management
VC money invested in startups: State and local legislators, funders and donors, external academic stakeholders, board management, research management
Number and size of consultancy agreements: State and local legislators, funders and donors, external academic stakeholders, board management, research management
Contract funding from industry: State and local legislators, funders and donors, board management, research management
Number of and list of new treatments: State and local legislators, board management, community, patients
Fraction of indirect costs covered: Funders and donors, board management, research management
Cost/benefit calculations: State and local legislators, funders and donors, board management, community, research management, patients
Broader metrics
Perceptions of equity, quality, access: State and local legislators, board management, community, research management, patients
Perceptions of staff: External academic stakeholders, board management, community, research management
Perceptions of community partners: State and local legislators, funders and donors, external academic stakeholders, board management, community, research management
Perceptions of external experts: State and local legislators, funders and donors, external academic stakeholders, board management, research management, patients
Perceptions of people participating in research: Board management, community, research management, patients
Attitudes of participants towards science and towards research: State and local legislators, funders and donors, board management, community, research management, patients
Narratives of success: State and local legislators, funders and donors, external academic stakeholders, board management, community, research management, patients
Narratives of performance: Board management, community, research management, patients
NOTE: Any metric could potentially be of interest to some members of any stakeholder group. This section gives an indication of the most likely interests and focus of each group.
Sources
Each metric draws on data collected through one or more sources. This section maps the metrics against the relevant sources that could be used to collect them.
Metrics for Assessing and Communicating the Value of Biomedical Research, by Potential Data Sources
Research impacts
Number of journal articles published: Bibliometric analysis
Number of citations: Bibliometric analysis
Number of research output downloads: Analysis of wider institutional records
Mentions in social media: Social media analysis
Number and size of grant awards: Analysis of research administrative documents, funder records
Number and size of awards from major funders: Analysis of research administrative documents, funder records, PI survey
Number of different research funders supporting research: Analysis of research administrative documents, funder records, PI survey
Success rate of applications: Analysis of research administrative documents
Catalogue of infrastructure: Analysis of research administrative documents, departmental secretary survey
Use of infrastructure by other researchers: Analysis of research administrative documents, departmental secretary survey
Measures of prestige
Number of editorships of high profile journals: Records of external academic bodies, PI survey
Number of staff on relevant boards and committees: Bibliometric analysis, PI survey
Number of academy members: Records of external academic bodies
Number and type of prizes: Records of external academic bodies, PI survey
Number of (international) speaker invitations/conference invitations: PI survey
Number of media mentions: Clippings service, PI survey
Number of applications per open post: Analysis of research administrative documents
Percentage of out-of-state and international applications per research job/PhD post: Analysis of research administrative documents
Track record of new hires: Analysis of research administrative documents, wider staff survey, staff interviews
Undergraduate applications: Analysis of wider institutional records
Grade Point Average of incoming students: Analysis of wider institutional records
Teaching and career development impacts
Grade Point Average of graduates: Analysis of wider institutional records, educational outcomes and workforce tracking data
Longitudinal data on career progression of students: Educational outcomes and workforce tracking data, alumni survey
Number of PhD graduates: Analysis of research administrative documents, educational outcomes and workforce tracking data
Completion rate of PhD graduates: Analysis of research administrative documents, educational outcomes and workforce tracking data
Number of publications per PhD graduate: Bibliometric analysis, analysis of research administrative documents
5/10/15-year career outcomes for PhD graduates: Educational outcomes and workforce tracking data, alumni survey
K to R conversion rate: Analysis of research administrative documents, funder records
Career outcomes for researchers: Educational outcomes and workforce tracking data, alumni survey
Subject coverage of the professional development programme: Internal policy documents
Uptake of the professional development programme: Analysis of research administrative documents
Feedback on the professional development programme: Feedback forms from events and courses
Improved educational attainment/reduced drop out rate: Educational outcomes and workforce tracking data
Research and institutional processes
Start-up time for research projects: Analysis of research administrative documents
Start-up time for clinical trials: Analysis of research administrative documents
Average time from funding to publication: Funder records
Proportion of funds spent on administration: Analysis of research administrative documents
Support staff to researcher ratio: Analysis of research administrative documents
Prompt payment of community partners: Analysis of research administrative documents, survey or interviews with community partners
How hiring decisions are made: PI interviews, representative case studies
How decisions are made to apply for grants: PI interviews, representative case studies
How publications decisions are made: PI interviews, representative case studies
Proportion of publications that are open access: Analysis of research administrative documents, PI survey
Proportion of trials where protocol and findings are published: Analysis of research administrative documents, PI survey
Description of institution's policy on health equity in research: Internal policy documents
Proportion of projects which consider health equity in their design and conduct: Analysis of wider institutional records, PI survey, PI interviews
Networks and dissemination
Number of collaborations on grant applications and projects: Analysis of research administrative documents, funder records, PI survey
Levels of co-authorship: Bibliometric analysis
Bibliometric networks: Bibliometric analysis
Total number of different collaborators across all projects: Analysis of research administrative documents, funder records, PI survey
Description of range of collaborations: Analysis of research administrative documents, PI survey
Number of research projects engaging community partners: Analysis of research administrative documents, PI survey
Number of research projects engaging community partners for the entire duration of the project: Analysis of research administrative documents, PI survey
Number of articles co-authored with community partner: Bibliometric analysis
Existence of specifically tailored material for different community groups: Analysis of research administrative documents, internal policy documents, PI survey
Size of communications office: Analysis of research administrative documents
Number of staff engaged in outreach: Analysis of research administrative documents, PI survey
Number of people attending outreach events and their perceptions: Analysis of wider institutional records, feedback forms from events and courses
Level of participation in clinical trials: Analysis of research administrative documents
Number of projects with an industry partner: Analysis of research administrative documents, PI survey
Industrial research funding for PhD/secondment positions in industry and PhD scholarships: Analysis of research administrative documents, departmental secretary survey
Number of policy secondments: Analysis of research administrative documents, departmental secretary survey
Policy impacts
Number of invitations from policy makers: Departmental secretary survey
Number of citations on clinical guidelines: Bibliometric analysis, review of key policy documents
Number of citations in policy documents: Review of key policy documents, PI survey
Health impacts
Improved health of patients: Quality improvement data, public health data, electronic health records data
Improved quality of care metrics: Analysis of wider institutional records, quality improvement data, electronic health records data
Number of lives touched: Analysis of wider institutional records, internal policy documents
Narrowing of health/health-care disparities: Public health data, electronic health records data
Improved awareness of preventative measures in the community: Survey or interviews with community partners
Number of treatment developed in house: PI survey
Number of new treatments available (adopted from elsewhere): Analysis of wider institutional records, PI survey
Percentage, number, and range of types of clinicians on research projects: Analysis of research administrative documents, PI survey, departmental secretary survey
Number of uses of research infrastructure in clinical practice: Analysis of research administrative documents, analysis of wider institutional records
Economic impacts and commercialization
Level of (local) spending: Analysis of research administrative documents, StarMetrics data
Amount of direct employment: StarMetrics data, labor market analysis
Number and type of new offices (including subsidiaries) in the area: Survey or interviews with companies, labor market analysis
Size of tech transfer office: Analysis of research administrative documents
Existence of Intellectual Property policy: Internal policy documents
Number of patent applications: Analysis of research administrative documents, funder records, patent database, PI survey
Number of patents awarded: Analysis of research administrative documents, patent database, PI survey
Number of patent citations: Patent database
Number of licensing agreements and licensing revenue: Analysis of research administrative documents
List/examples of know-how taken up by industry: PI survey, high impact case studies
Number of private sector innovations/products/devices brought to market: Survey or interviews with companies
Number of spin outs: Analysis of research administrative documents, departmental secretary survey
VC money invested in startups: Analysis of research administrative documents
Number and size of consultancy agreements: Analysis of research administrative documents
Contract funding from industry: Analysis of research administrative documents
Number of and list of new treatments: Analysis of wider institutional records, wider staff survey
Fraction of indirect costs covered: Analysis of research administrative documents
Cost/benefit calculations: Analysis of research administrative documents, quality improvement data, public health data, electronic health records data, StarMetrics data, labor market analysis
Broader metrics
Perceptions of equity, quality, access: Survey or interviews with research participants, survey or interviews with community partners
Perceptions of staff: PI survey, wider staff survey, PI interviews, staff interviews
Perceptions of community partners: Survey or interviews with community partners
Perceptions of external experts: External researcher site visits or peer review
Perceptions of people participating in research: Survey or interviews with research participants
Attitudes of participants towards science and towards research: Survey or interviews with research participants
Narratives of success: High impact case studies
Narratives of performance: Representative case studies
NOTE: This section does not represent an exhaustive list, but rather highlights the potential data sources that stand out as most appropriate for each metric.
Type
Each metric has been classified into a particular type. The nine types identified and captured in Figure 1 cover the pathway from research production to research impact. Each type represents a different aspect of this pathway. The nine types are as follows:
Research impacts (Research)—metrics that capture the direct development and outputs arising from conducted research
Measures of prestige (Prestige)—metrics that reflect external recognition of quality of conducted research
Teaching and career development impacts (Career)—metrics that capture the teaching and career development of researchers
Research and institutional processes (Process)—metrics that capture the effectiveness and efficiency of the administrative and institutional processes underlying research
Networks and dissemination (Network)—metrics that capture the interactions of researchers and the academic institution with external stakeholders
Policy impacts (Policy)—metrics that capture the changes in policy to which research conducted at the institution has contributed
Health impacts (Health)—metrics that capture the changes in health outcomes to which research conducted at the institution has contributed
Economic impacts and commercialization (Economics)—metrics that capture the changes in the (local) economy to which research conducted at the institution has contributed
Broader metrics (Broader)—approaches that can capture information across a range of these categories.
Considerations for Good Use of the Metrics
The classifications provided can be used to guide institutions as they select metrics and evaluation approaches for measuring their performance. To provide a balanced evaluation, an academic medical center is likely to want to select a range of metrics and types that cover various stakeholder groups, while also drawing on the data sources already available. The two classification sections aim to enable institutions to match metrics to their interests and available data.
It is important to note that not all metrics are appropriate to all fields of medical research. For example, some fields are more likely to produce patents than others. In addition, metrics cannot always be compared across fields. For example, what counts as a large number of publications in one field might seem a small amount to another. Fields and their differences should therefore be given substantial consideration in the selection of metrics.
Furthermore, the maturity and stage of development of a project or institution will determine whether or not some metrics are appropriate. Newer projects or institutions may not yet have produced long-term societal impacts. Metrics capturing such impacts would be inappropriate to the early stages of development, but could be introduced later in time.
The research described in this study was conducted by RAND Europe in collaboration with the Association of American Medical Colleges.
1 S. Guthrie, W. Wamae, S. Diepeveen, S. Wooding, and J. Grant, Measuring Research: A Guide to Research Evaluation Frameworks and Tools, Santa Monica, Calif.: RAND Corporation, MG-1217-AAMC, 2013.
2 A. Bonham and P. M. Alberti, “From Inputs to Impacts: Assessing and Communicating the Full Value of Biomedical Research,” Academic Medicine, forthcoming.