Metascience Novelty Indicators Challenge
A global initiative to identify and develop measures that can automatically identify novelty in research and explore pathways to application in research evaluation and policy
What is the issue?
Photo by flucas/Adobe Stock
Novelty is an important feature of scientific research, one that can indicate its future contributions to scientific progress, knowledge advancement, and societal impact. In a recent survey of ‘Research leaders’ published in the journal Nature, ‘novel, innovative and original’ findings were considered primary drivers of high academic impact. And whilst novelty is only one of many complex dimensions that contribute to the quality of scientific research, identifying novelty early in the research lifecycle can offer substantial value to the research ecosystem.
Recent progress in scientometrics research offers the potential to create tools that can identify novelty automatically at the time of publication. Such tools both build on ‘classic’ bibliometric approaches, such as reference networks, but are also increasingly able to utilise large language models and AI to process the full text of research.
How are we helping?
RAND Europe, together with partners at the Science Policy Research Unit (SPRU) at the University of Sussex Business School, Challenge Works, and prophy.ai, led the delivery of the Metascience Novelty Indicators Challenge.
Delivered as a Challenge Prize, the competition launched in September 2025 and brought together data scientists, researchers, industry practitioners, and academics from a range of fields and sectors, who were invited to submit tools that identified novelty in academic publications. These were then judged on their ability to achieve the highest level of accuracy towards a dataset of human expert novelty scores.
To judge entrants, SPRU and RAND Europe worked to establish a comparator dataset of over 40k human expert raters, involving a corpus of close to 38k publications taken from OpenAlex, making it one of, if not the largest, datasets of its kind. You can read more about the challenge and the challenge winners on the Challenge website or in the Science article.
To extract the most value out of the challenge outcome, RAND Europe is leading a policy analysis and engagement process that will explore how these tools could contribute to and aid research evaluation and funding systems. We are conducting a targeted evidence review examining how novelty is currently perceived and assessed in R&I policy and practice, the effectiveness of existing approaches and barriers to adoption. Upcoming policy workshops will explore both the transformative potential of employing novelty indicators and risks: applicability across disciplines, mitigating potential perverse incentives and ensuring fairness and legitimacy.
