Designing an evaluation framework for the NHMRC

clockwork evaluation wheel

EtiAmmos/Adobe Stock

The National Health and Medical Research Council (NHMRC), Australia, have established a new grant program meant to emphasize fairness, reduce burden, and boost innovation and creativity in research funding processes. Researchers developed an evaluation framework and set of metrics to monitor the performance of the program.

Background

The National Health and Medical Research Council (NHMRC) is the Australian Government's primary health and medical research funding agency, investing in research across four main areas of health and medical research.

Following a review of its grant program in 2016, the NHMRC decided to adopt a new grant program. To assist with the transition to and implementation of this new program, the NHMRC commissioned RAND Australia to design and develop an evaluation framework to assess performance against its aims to:

  1. Encourage greater creativity and innovation in research.
  2. Minimise the burden on researchers of application and peer review so that researchers can spend more time producing high quality research.
  3. Provide opportunities for talented researchers at all career stages to contribute to the improvement of human health.

Goals

RAND Australia and RAND Europe researchers conducted a review of international practice in program evaluation, with the aim of building an understanding of the methods and measures available for evaluating against desired outcomes, as well as developing an overview of the underpinning principles and challenges in program evaluation, and how these can be addressed.

Using data collected on the NHMRC's data and grant management systems, the study developed an evaluation framework and set of metrics which can be used to monitor the performance of the NHMRC's new grant program over time.

Methodology

The study was conducted using a variety of methodologies, including a rapid evidence assessment, key informant interviews, and a review of the NHMRC's data and grant management systems.

Findings

Learnings from our review of international practice

  • Our review of international practice found that efforts to systematically measure opportunity across all career stages, as well as burden, innovation and creativity in the grant funding process, have so far been limited, but there are examples on which we can build.

We have developed a framework and a set of practical measures to assess the performance of NHMRC's new grant program

  • Metrics developed build on existing data collected but also span new sources and some are more forward looking – for example, the use of text mining and potential development of new surveys. Some metrics can be implemented straight away and others can be introduced as the grant program matures.