Context and Evaluation Aims
The COVID-19 pandemic has had an unprecedented impact on populations across the globe. As of 15 November 2021, over 9.5 million positive COVID-19 cases and over 165,000 related deaths were recorded in the United Kingdom (UK) alone (UK Government, 2021). The ability to identify individuals infected with the SARS-CoV-2 virus, which causes COVID-19, and to sequence and understand the variants of the virus that have been circulating in the UK since the onset of the pandemic have been vital in informing public health decision making and efforts to control its spread. The work of pathogen genomics experts who are part of the COVID-19 Genomics UK (COG-UK) Consortium has underpinned key sequencing and research efforts.
The COG-UK consortium was established soon after the UK went into its first lockdown of the COVID-19 pandemic in March 2020. On 1 April 2020, COG-UK received approximately £20 million in funding from the National Institute for Health Research (NIHR), the Medical Research Council (MRC)—part of United Kingdom Research and Innovation (UKRI; grant reference number: MC_PC_19027)—and Genome Research Limited (operating as the Wellcome Sanger Institute). In January 2021 and April 2021, COG-UK received an additional £11.6 million from the Testing Innovation Fund and £5 million from the Department of Health and Social Care Test and Trace, respectively, to bolster their sequencing output. COG-UK is a collaborative effort between 16 academic institutions, the UK's four public health agencies (PHAs) of England, Scotland, Wales, and Northern Ireland, the Wellcome Sanger Institute, four Lighthouse Labs1 and 79 National Health Service (NHS) Trusts or other organisations. COG-UK builds on the UK's strengths in pathogen genomics, population health sciences and health informatics. The consortium was set up with the aims to2:
- Provide data, analysis, tools, and research that can help guide public health decision making and policy relating to the COVID-19 pandemic;
- Advance understanding of genetic changes in the SARS-CoV-2 virus and how they relate to the spread of the virus and severity of COVID-19 symptoms, all of which matter for public health decision making and the development and evaluation of treatments and vaccines; and
- Support national research studies, including those that can help enable future evaluations of the effectiveness of various pharmacological and non-pharmacological interventions to prevent or treat COVID-19.
Given growing realisation of the importance of pathogen genomics for public health, the scale of investment made, and the commitment to widely sharing learning from COG-UK's experience, the consortium commissioned the not-for-profit institute RAND Europe to evaluate and learn about COG-UK's progress, evolution and impacts. More specifically, the evaluation set out to:
- Examine COG-UK's delivery against its aims in terms of its outputs, outcomes and impacts;
- Understand how processes related to governance, management and operations impacted delivery;
- Learn about enablers of progress as well as challenges experienced;
- Provide valuable formative learning for any potential future consortium phases and/or for other related efforts, including sustainability and legacy.
The evaluation adopted a mixed-methods approach. This involved:
- Developing an evaluation framework based on specifying a COG-UK theory of change (i.e. an understanding of what the consortium set out to achieve and how) and associated evaluation indicators;
- Collecting and analysing self-reported qualitative and quantitative data on evaluation indicators;
- Conducting in-depth semi-structured interviews with diverse stakeholders involved with COG-UK; and
- Undertaking cross-analysis, synthesis and reporting.
Data was collected between February 2021 and October 2021. Two workshops were conducted in February 2021 to specify COG-UK's theory of change and evaluation framework. Self-reported data on COG-UK's evolution, outputs, outcomes and impacts were collected for the evaluation period 1 March 2020 to 31 July 2021. Data were provided to RAND Europe in September 2021 and in-depth semi-structured interviews were conducted between April and October 2021.3
Although this evaluation provides deep and rich insights about COG-UK's experiences, there are some limitations to consider when interpreting its findings.
First, the evaluation began after the consortium had been operating for eight months. Thus, the findings presented were collected retrospectively. This means that some elements—such as developing the theory of change—depended on a degree of participants' recollection of experiences in the consortium's earlier phases.
Second, this evaluation's scope and resources meant that the focus was primarily on understanding COG-UK's delivery against its aims, identifying the factors shaping its performance, and discerning relevant lessons for the future. COG-UK unfolded in an environment without any "business as usual" or control groups for comparison.
Consequently, this evaluation discusses COG-UK's contributions to the public health landscape and pandemic response, but there is no counterfactual from the UK to assess this against. Future evaluations may enrich this inquiry, perhaps considering comparators and learning from international experiences of other pathogen-genomic sequencing initiatives.
A further caveat is that an audit of the self-reported data COG-UK provided as part of this evaluation was outside the scope of this work. However, the evaluation team is confident that the diverse perspectives and experiences shared through the in-depth interviews and the specificity of the self-reported data support an objective and comprehensive analysis of COG-UK outputs and impacts as well as a detailed understanding of the diversity of influences on the consortium's evolution and impacts.
In the following sections, we overview key insights and learning and their implications and lessons of relevance to future efforts.
COG-UK's Contributions to Understanding and Responding to the COVID-19 Pandemic
COG-UK has made diverse contributions to understanding and responding to the COVID-19 pandemic. The core outputs, outcomes and impacts of COG-UK activity are briefly summarised below and elaborated on in the narrative that follows.
Key COG-UK Achievements
- The consortium has helped advance scientific knowledge about SARS-CoV-2 and helped improve methodologies that can support high quality and efficient sequencing and pathogen genomics research and analysis.
- COG-UK has also provided data and analytics that have informed key policy and public health decisions made in response to the COVID-19 pandemic in the UK.
- The sequencing and analysis of SARS-CoV-2 genomes and the linkage of genomics data to epidemiological and patient outcomes data by COG-UK partners have informed medical innovation efforts, including research and evaluations of vaccine efficacy against specific variants of SARS-CoV-2 and research on the susceptibility of viral variants to therapeutics against COVID-19.
- COG-UK's data, research analytics, and dissemination efforts have also influenced how decision makers in the UK value and view the field of pathogen genomics, as a partner in building effective public health systems.
- COG-UK's resources and activities have strengthened capacity for pathogen genomics, which, if sustained, has the potential to significantly bolster the UK's ability to prepare and respond to future infectious disease threats.
- The consortium's impacts extend beyond the UK's borders. COG-UK's approach to supporting pathogen-genomics sequencing, research and analysis, and learning from COG-UK's experience has also influenced international SARS-CoV-2 sequencing initiatives.
1. Contributions to Scientific Knowledge About SARS-CoV-2 and to Advancing Methods That Support Sequencing and Pathogen Genomics Research
The sequencing of viral genomes has been essential to research and analysis efforts to understand the SARS-CoV-2 virus and its behaviour. COG-UK sequenced over 800,000 SARS-CoV-2 genomes across the UK between 1 April 2020 and 31 July 2021.4 The consortium increased sequencing capacity at its sites from 5,000 to 30,000 samples per week throughout the evaluation timeframe and reduced the average cost of sample sequencing by approximately 30 per cent (from £56 to £40 blended per sample).5
Consortium members' research and analyses have advanced knowledge about the SARS-CoV-2 virus regarding variants of concern, viral behaviour, transmissibility and spread, and the impact of various public health measures. To achieve this, the consortium has been committed to openly sharing its findings and making them freely accessible since its inception. During the evaluation period for which self-reported data was provided,6 COG-UK partners produced 53 publications (including 51 academic papers and 2 non-academic reports), contributing to insights that helped inform the pandemic response. COG-UK partners shared this knowledge widely through participation in 42 conferences, seminars or training events on SARS-CoV-2 genomics.7 COG-UK has also made its data widely and freely available through public-domain databases, e.g. the Global Initiative on Sharing Avian Influenza Data (GISAID), European Nucleotide Archive (ENA), and New and Emerging Respiratory Virus Threats Advisory Group (NERVTAG).
In addition, the software tools, sequencing and analysis protocols developed by COG-UK researchers have improved methods available for public health genomics, including those beyond the COVID-19 pandemic. Similarly, COG-UK partners' efforts to help link viral sequencing and patient metadata (e.g. vaccination records, clinical, demographic and postcode information and travel history) and viral and host genome data are building a critical resource for further research of public health significance. The existing UK Cloud Infrastructure for Microbial Bioinformatics (CLIMB) data and computing infrastructure held by public health partners8 was leveraged for COG-UK activities and has played a vital role in these efforts.
While these contributions are notable, the pace and scale of COG-UK outputs and contributions to scientific knowledge and methods underlying pathogen genomics will not be straightforward to sustain and apply to future public health challenges. The ability to do so will depend on securing long-term resources to support the required public health workforce and relationships between academic organisations, NHS sites and PHAs across the UK within conducive governance and management arrangements.
2. Informing Key Policy and Public Health Decisions in the UK's Pandemic Response
COG-UK's sequencing data and analyses have helped identify variants of significance circulating in the four UK nations. The consortium's work informed policies related to border control, travel, lockdown and social distancing and improved policymaker and public understanding of links between new variants and disease severity. The consortium's pathogen sequencing has also impacted decision making in local settings, including hospitals, care homes and universities by helping to understand sources of outbreaks and transmission patterns and informing infection prevention and control and patient-safety reporting.
From the outset, COG-UK has worked closely with public health decision makers to maximise the value and impact of SARS-CoV-2 genomic data on public health. They achieved this through several routes, including: 1) participation on various committees and working groups, e.g. in Department of Health and Social Care (DHSC), Public Health England (PHE), Scientific Advisory Group for Emergencies (SAGE), and NERVTAG, 2) contributions to 36 external reports produced by policy and public health decision makers during the evaluation timeframe, 3) direct reporting to policymakers (e.g. 18 COG-UK reports to SAGE), and 4) provision of informal feedback. Throughout the pandemic, COG-UK has worked in partnerships with actors in the public health research landscape to connect sequencing and patient metadata further and help inform future research studies and public health decision making.
The consortium supported SARS-CoV-2 genomic sequencing needs across England, Wales, Scotland and Northern Ireland using tools such as sequencing coverage reports9 to help prioritise samples for sequencing and analysis. Despite significant contributions to identifying variants of concern across the four nations, COG-UK's ability to inform regional decision making in a timely manner was occasionally hampered by factors outside its direct control. Such factors included limited PHA access to the patient metadata needed to help prioritise localities from which samples should be sequenced and understand relationships between outbreaks and travel histories or vaccination records. Some of the stakeholders interviewed during this evaluation also noted possible differences in the degree of influence different PHAs had on prioritising samples for sequencing. There is an opportunity for PHAs and other partners to work together to further bolster sequencing capacity within the devolved nations in the future.
3. Informing Medical Innovation: Testing Vaccine Efficacy Against Specific Variants of SARS-CoV-2 and Better Understanding Therapeutics
COG-UK has also provided important inputs into medical-innovation responses to the pandemic through its data and analytics. For example, the consortium's open sharing of data and insights have informed assessments of vaccine efficacy against different viral variants. COG-UK is also working with various consortia to identify and characterise variants of concern to inform the development of the next generation of vaccines. The consortium's data on and analysis of viral mutations have helped to better understand potential treatments such as convalescent plasma therapy. As the consortium progresses its efforts to link host and viral genome data and scales-up data linkage for viral and patient metadata, there will be opportunities for further engagement with those developing and evaluating vaccines and therapeutics for COVID-19.
4. Changing How Decision Makers View and Value Pathogen Genomics
COG-UK has had a significant impact raising awareness about the importance of pathogen genomics as a discipline. It has increased policymaker appreciation of pathogen-genomics' value, as well as increased demand for the use of sequencing insights in public health decision making as part of the COVID-19 response. However, it remains to be seen whether increased recognition of pathogen sequencing's importance for pandemic preparedness and response will translate into increased investment and capacity for its use in other areas. Such areas might include surveillance of other infectious disease threats and public health challenges such as antimicrobial resistance where pathogen sequencing can inform medical innovation. In spring 2021, COG-UK began transitioning routine sequencing capacity for SARS-CoV-2 from research institutions to PHAs. Sequencing sites in universities and research institutes are providing a safety net for potential surges in sequencing capacity needs. It will be important to nurture and grow this capacity in PHAs over time.
5. Capacity for Pathogen Genomics in the UK
COG-UK has made diverse contributions to bolstering pathogen genomics sequencing and research capacity in the UK. More specifically:
- Strengthening the workforce: COG-UK has trained up staff across different professions (e.g. healthcare professionals, researchers, PHA staff), career-development stages and UK locations to improve sequencing, analysis and interpretation skills. The consortium estimates having trained over 800 individuals across the UK during the evaluation timeframe.
- Physical and data infrastructure improvements: COG-UK purchased sequencing equipment and implemented software tools that are now available at 17 sequencing sites across the UK,10 providing newly increased capacity. Building upon the existing CLIMB data infrastructure, COG-UK has also used its funds to rapidly develop additional computational infrastructure (e.g. CLIMB-COVID).
- Leadership, management and governance arrangements: The consortium established a governance structure representing the diverse geographies and stakeholders involved in the UK public health landscape. It also established contractual arrangements, operational protocols and legal frameworks to support a four-nation, multistakeholder approach. COG-UK's collaborative approach has demonstrated the importance of conducive governance and management support for networked public health efforts and of aligning such governance with pre-existing institutional practices. Whether such approaches can be applied and sustained beyond the COG-UK collaboration will determine this approach's feasibility for responding to future public health threats.
- Strengthened relationships between key organisations in the public health landscape across the UK: As a consortium of academic research partners, the Wellcome Sanger Institute, PHAs, as well as collaborators including Lighthouse Labs and NHS foundations and trusts, COG-UK has fostered collaboration between diverse professions in the interest of advancing science and informing policy. If sustained, a model of close interaction between researchers and PHAs across the devolved nations may offer a new paradigm for the future of UK public health.
Investments in equipment, methodological development and workforce skills have also supported reductions in sequencing costs per sample and improvements in sequencing and result-reporting turnaround times. On average, time from sample collection to sequencing-data upload decreased by 70 per cent across consortium sites, from 20 days in April 2020 to 6 days in June 2021.11
6. Impact on the International Pandemic-Response Effort
COG-UK's data and resources are available for use by the international community. Consortium members have also advised some other countries on their sequencing and data-sharing strategies (e.g. Canada, France, Israel, the United States [US]), supported pathogen genomics efforts in 17 low-and-middle-income countries12), and shared expertise through roles in international working groups and councils (e.g. the World Health Organisation [WHO] working groups and the Global Early Warning System Action Collaborative Advisory Council).
COG-UK's experience offers lessons directly relevant to future pandemics and public health threats, with some likely to have international relevance. Examples include learning about the critical role of pathogen genomics in supporting a rapid response to pandemics and the methodological tools and protocols to use in sequencing, In addition, learning related to the logistics of collaboration between different stakeholders and the coordination of various actors may have elements applicable to diverse geographies. However, the consortium's primary focus on UK-based data may limit internationally applicable insights. This aspect merits further research to understand which aspects of COG-UK's structure and function may be adaptable to other contexts and which are more context-specific.
Influences on COG-UK's Evolution and Impacts
Any effort to learn how a networked and multi-stakeholder public health genomics approach can support responses to infectious disease pandemics benefits from understanding the diversity of factors that influenced COG-UK's efforts. The consortium's evolution and impact have been affected by features internal to its structure and operations, and by a rapidly evolving and unpredictable external environment regarding both the virus evolution and evolution in the policy landscape. More specifically, key influences related to:
- The ability to mobilise and sustain individual and institutional commitment to consortium activities, including a commitment to rapid delivery and responsiveness to increasing demand for sequencing activities. To a large extent, this depended on individual and institutional goodwill. It was also influenced by leadership, governance and management structures and actions and by substantial investments of time and efforts to nurture productive relationships and interactions between diverse COG-UK members (see Table 1).
- The resource environment—including financial resources, physical and data infrastructure and connectivity and human resources. These were essential in supporting COG-UK's delivery at pace and scale (see Table 2). Critical factors enabling COG-UK to be "fleet on its feet" included timely access to substantial funding and the ability to rapidly onboard UK sites across academic, public health and NHS organisations and redirect staff capacity toward responding to the pandemic. Access to pre-existing data, computing and cloud infrastructure was also critical. Advocating for the need for public health genomics sequencing by leaders who came up with the idea to establish COG-UK was key to securing financial support that allowed for mobilising staff resources and requisite physical and data infrastructure in a timely manner. However, the pace and scale of activity and the demands on staff were considerable; they will be challenging to sustain in the absence of a long-term resourcing strategy for the future.
- The ability to navigate external environmental forces, particularly those related to the speed and unpredictability with which the COVID-19 pandemic unfolded. Both the unpredictability of viral evolution in terms of variants and their transmissibility, and evolution in the policy landscape impacted on how COG-UK carried out its activities. The urgency of the pandemic, coupled with COG-UK's resolute management, strategy, processes, partner commitment and goodwill, helped focus consortium members on the most urgent tasks and enabled agility. However, it also required continuous operation in "fire-fighting" mode, which is unlikely to be sustainable for the longer term (see Table 3).
Table 1. An Overview of Enablers and Challenges Related to COG-UK's Ability to Mobilise and Sustain Member Commitment to Consortium Activity
|Influence: The Ability to Mobilise and Sustain Individual and Institutional Commitment to Consortium Activities|
|Key Enablers||Key Challenges|
|Individual and institutional goodwill:
Supportive leadership, governance, and management:
Overall productive relationships and interactions in the COG-UK network:
|Challenges in matching individual and institutional capacity to demand:
Governance and management challenges:
Relational challenges in an inherently complex and diverse network:
Wider political developments:
Table 2. An Overview of Enablers and Challenges Related to COG-UK Resources
|Influence: Resources—Funding, Physical and Data Infrastructure, Human Resources|
|Key Enablers||Key Challenges|
|Financial resource support:
Physical and data infrastructure:
|Financial resource challenges:
Physical and data infrastructure challenges:
Human resource challenges:
Table 3. An Overview of Enablers and Challenges Related to COG-UK's Ability to Adapt to Unpredictable Conditions Associated with the COVID-19 Pandemic
|Influence: The Ability to Respond to the Urgent and Unpredictable Nature of the Pandemic|
|Key Enablers||Key Challenges|
In Reflection and Looking to the Future
COG-UK has made a significant and valuable contribution to the UK's public health genomics landscape. However, what needs to be sustained is not necessarily the network as it operated during the evaluation timeframe, but the ecosystem that has been built around it.
Reflecting on the learning gained and looking to the future, COG-UK's legacy will depend on decision makers' abilities to:
- Deliver public health genomics capacity guided by a clear, prioritised, long-term strategic plan: Priorities will need to reflect and reconcile the interests of the scientific community, citizens and patients, and be aligned to the long-term priorities of governments and public health decision-makers across the four nations of the UK.
- Maintain momentum, motivation and goodwill to support a network that can bring together diverse organisations across the four nations without over-reliance on goodwill alone: Long-term funding and sustaining committed leadership will be critical. A workforce development strategy that considers novel career pathways in PHAs and academic settings will be needed, alongside an existing or novel convening structure that can ensure a coordinated national approach as well as respond to the devolved nations' unique local needs.
- Ensure the involvement of all relevant actors: COG-UK mobilised the engagement of researchers, PHAs and NHS sites across the country. As the consortium enters the next phase of its existence or morphs into a legacy structure, it may need new expertise, e.g. bringing in private-sector partners to link genomics research with medical innovation, international expertise, patient-and-public engagement and additional involvement from the NHS to extend the role of pathogen genomics in the NHS.
- Stabilise and ensure adequately funded governance, management and administrative arrangements to support networked pathogen genomics capacity in the UK: Attention should be given to where, and how far, elements of COG-UK governance that enabled rapid delivery, minimal bureaucracy and novel practices co-existing with established institutional governance and management systems, may be adaptable to future efforts. Academic researchers will require requisite independence but governance must also support synergies and ensure a shared sense of purpose across research and sequencing services informing public health.
- Advance data linkage in the public health landscape: Access to linked data sets will be fundamental to understanding the relationship between infectious-agent genetics and behaviour on the one hand and disease severity and patient outcomes on the other. It will also underpin efforts to inform the development and evaluation of medical innovations. Wider collaboration between actors in the UK's public-health and health-data landscape will be needed.
- Ensure a sustainable division of labour between diverse stakeholders in the public health genomics landscape: Attention must be paid to ensure the sustainability of the workforce required to service routine sequencing needs in PHAs and to ensure that trained research talent is not lost from academic settings. Considering what may be different in terms of workforce needs if COG-UK (or its legacy) were to tackle other infectious diseases or public health challenges will also matter.
- Revisit the UK's role in the global pathogen genomics landscape: COG-UK members' expertise impacted international public-health genomics efforts and there is further potential to develop COG-UK as a global training resource and expertise-sharing network. At the same time, COG-UK is largely built on UK data, and a future legacy effort would benefit from an explicit focus on integrating international experiences and embeddedness in coordinated global efforts.