What Is a High-Risk AI System and Why It Matters
This section defines high-risk AI systems based on how the model is used and what it can do and summarizes the types of harms these systems can cause.
What Is a High-Risk AI System?
Systems are considered high risk when they
- pose significant threats to health, safety, civil rights, or economic opportunity
- are a critical component in safety-sensitive or consequential systems
- play a substantial role in high-stakes decisionmaking
- centralize or automate capabilities that, if misused or incorrect, could produce severe or irreversible harm (e.g., chemical design, biological modeling, industrial optimization, physical controls).
Classifying Risk: Context and Capability
High risk can be assessed along two complementary perspectives: context-sensitive and capability-aware.
- Context-sensitive: The same model can be low risk when used to summarize public news articles but high risk when used to analyze a patient’s medical symptoms and recommend a diagnosis. This definition focuses on how the model is used as opposed to inherent attributes (e.g., latent capabilities) of a model.
- Capability-aware: Definitions should consider the model’s raw capabilities, which can be misused regardless of the model’s intended use (e.g., research models repurposed to design toxic agents). This “dual-use” perspective suggests that risk is intrinsically tied to capability and that access (e.g., open-weight models) expands the attack surface, and thus risk.
In practice, these two approaches to defining high-risk AI systems—context-sensitive and capability-aware—are complementary and mutually reinforcing. Furthermore, context and capability interact:
- High-risk systems often appear when a capable model is deployed in a sensitive environment—clinical workflows, autonomous systems, software supply chains, chemical and biological research, identity verification, or large-scale decisionmaking.
- Risk also arises when less sensitive environments integrate models with tools, data sources, or automation. Access to external actions (e.g., running code, querying databases, operating equipment) or to sensitive information (e.g., proprietary datasets, regulated materials, confidential user data) expands both the model’s leverage and the potential blast radius of failure.
- Finally, high-risk conditions occur when a model is internally deployed but encodes specialized scientific, operational, or domain-expert knowledge that can enable harmful actions even without broad system integrations. For example, a materials-optimization model may generate unsafe protocols or misleading designs that carry real-world consequences, even when isolated behind enterprise controls.
As models become more interconnected—with plugins, retrieval systems, agents, and autonomous tools—the boundary between “model risk” and “systems risk” blurs, creating new opportunities for misuse or accidental harm.
Why High-Risk AI Systems Matter
Advanced AI models and the systems built around them matter because failures in these environments can produce rapid, large-scale, and sometimes irreversible harm. The more capable a model is, and the more integrated it is into operational workflows, the greater the consequences of something going wrong. These risks apply broadly to all models capable of influencing decisions, generating actions, or interacting with sensitive data, tools, or physical processes.
Importantly, risk is not limited to general-purpose language models. Internally deployed or highly specialized models can generate harmful or dual-use outputs even in isolation, with no external tools, plugins, or user-facing interfaces.
The ways in which advanced or bespoke AI models centralize and operationalize sensitive capabilities at scale amplify risks in numerous ways, including
- deliberate misuse (e.g., to generate malware or disinformation)
- accidental production of harmful content (e.g., due tobecause of hallucination, bias, flawed reasoning)
- subversion or exploitation by attackers (e.g., using indirect prompt injection, model hijacking, supply chain attacks)
- rapid scaling (i.e., models can scale harm rapidly due tobecause of automation, accessibility, and function)
- irreversibility of harm (e.g., AI-driven actions, such as biological design, chemical synthesis, or infrastructure manipulation, cannot be undone once information is leaked or an action is taken)
- domain-specific hazards (e.g., models generating high-risk optimization strategies, toxic molecules, or misleading scientific predictions that influence downstream experiments or physical systems)
- tacit-knowledge risks (e.g., where scientific or operational models surface actionable, step-by-step procedures that lower barriers to harmful activities).
Additionally, consequential risks emerge when models are deployed as part of broader AI systems, such as the pipelines that combine models with data, retrieval components, memory, automation, or tool execution capabilities. In these composite environments, a small vulnerability can have an outsized impact. High-risk models, therefore, matter not only because of their system integrations but also their inherent ability to generate dangerous knowledge, influence sensitive decisions, or shape real-world actions. As AI capabilities continue to advance, the need to secure both high-risk models and the systems around them becomes even more critical.