Examining risks and response for AI loss of control incidents
Photo by Looker_Studio/Adobe Stock
What is the issue?
As artificial intelligence (AI) systems become increasingly embedded in essential infrastructure and services, the risks associated with unintended failures rise. Future critical failures from advanced AI models could trigger widespread disruptions across essential services and infrastructure networks, potentially amplifying existing vulnerabilities in other domains. Developing comprehensive emergency response protocols could help mitigate these significant risks. This report focuses on understanding and addressing a specific class of such risks: AI loss of control (LOC) scenarios, defined as situations where human oversight fails to adequately constrain an autonomous, general-purpose AI, leading to unintended and potentially catastrophic consequences.
How did we help?
The research started with a rapid evidence assessment (REA) of research on AI LOC and emergency response in cybersecurity and biosafety. Literature on AI LOC – particularly regarding prevention, preparedness and response – is extremely limited. However, cybersecurity and biosafety offer useful analogies, providing insights into governance mechanisms, coordination challenges and best practices for managing high-risk, high-uncertainty crises.
The research also engaged in scenario development and analysis, with the team constructing a non-realised scenario, in which a potential LOC incident is detected and mitigated, and a realised scenario, in which AI developers and users can no longer constrain the model’s function, resulting in severe and unintended harm. The team then assessed existing policies and frameworks; identified response, prevention and preparedness strategies; and highlighted key coordination challenges. Finally, the team developed recommendations for improving LOC prevention and response, based on insights from the prior phases.
What did we find?
LOC risks are increasingly plausible and remain unaddressed
Researchers have identified warning signs of control-undermining capabilities in advanced AI models – including deception, self-preservation and autonomous replication – which could potentially enable increasingly capable models to evade human oversight.
Photo by royyimzy/Adobe Stock
Detection and early warning challenges
Governments and other stakeholders lack a common framework to analyse and respond to LOC risks. There is no clear consensus on which AI capabilities could lead to LOC, how safeguards may interact with such capabilities, or the best warning signs of LOC risks. This fragmented understanding hampers the ability of model developers or governments to detect early LOC warnings. Furthermore, current detection methods rely on pre-deployment model evaluations and ongoing monitoring by AI developers, with limited validation by independent third-party evaluators.
However, models may operate differently in testing environments, potentially interacting with deployment contexts in unexpected ways. Open-source models present challenges to detection given the potential for unmonitored access and modifications to the model with limited oversight.
Escalation gaps
Safety frameworks published by industry have yet to align on a consistent approach to risk escalation. Importantly, there are no clear thresholds for when a LOC incident should trigger an emergency response.
Containment and mitigation limitations
Containing a LOC event requires advances in technical AI safety. Traditional cybersecurity safeguards such as endpoint detection, firewalls and malware detection are essential but may be insufficient. Containment measures may be ineffective if AI systems gain significant control over resources before risks are detected.
What can be done?
Detection of LOC threats:
Governments, with AI developers and other stakeholders, should establish a clear, shared definition of AI LOC and a set of criteria for detection.
AI developers and researchers should refine detection by developing standardised benchmarks and improving their reliability and validity.
Governments should enhance awareness and information sharing between all stakeholders, including the tracking of computer resources.
Actions for escalation:
AI developers should establish well-defined escalation protocols and conduct regular training exercises to ensure their effectiveness.
Government stakeholders should consider mandatory reporting mechanisms for AI risks and potential incidents.
Government stakeholders should establish disclosure channels and whistleblower safeguards for employees of AI developers.
AI developers, AISIs and relevant government departments should enhance cross-sector and international coordination.
Actions for containment and mitigation:
AI developers should prepare containment measures that are rapid and flexible.
AI developers and other stakeholders should further explore and advance research on containment methods.
AI developers, external researchers and AISIs should prioritise safety and alignment measures, including by building validated safety cases.
Government stakeholders should seek to strengthen AI security to protect model weights and algorithmic techniques.
Governments and developers should improve safety governance by fostering robust safety cultures and adopting secure-by-design principles.