Limiting Execution Depth and Incorporating Human Oversight

AI systems with autonomous or tool-using capabilities—especially LLMs—are increasingly capable of autonomously using tools and executing code to carry out multistep tasks. This capability introduces risks, such as AI systems recursively calling themselves, creating unbounded positive feedback loops, or performing tasks without the appropriate permissions. This control focuses on limiting the autonomous capabilities of LLMs by bounding their execution depth and incorporating human oversight for specific tool use. These measures help maintain control over recursive and multistep planning, reducing the likelihood of unintended outcomes.

To address these risks, one should consider such strategies as using container resource limits to prevent runaway processes;⁠1 setting strict CPU, memory, and disk usage limits for AI agent processes;⁠2 implementing timeout mechanism operations; and establishing manual override capabilities to immediately halt AI agent execution.⁠3 Employing a modular design—breaking down complex tasks into smaller, independent subagents with limited scopes—can also enhance manageability and control.

FocalX, an AI-powered automotive inspection firm, describes human-in-the-loop (HITL) systems as those in which autonomous algorithms operate under human supervision. In these systems, humans set goals, monitor performance, and intervene when necessary to ensure that AI aligns with intended outcomes and ethical standards. For example, advanced driver assistance systems can autonomously accelerate, brake, and steer vehicles, but drivers are expected to remain on standby and take control as needed.

The Future Society offers several recommendations for implementing HITL mechanisms, including setting checkpoints where human authorization is required within agents’ workflows and incorporating a permission management system.⁠4 This system should include explicit declarations of required permissions in the agents’ configuration, as well as dynamic permission requests at runtime for sensitive operations. See Table A.1.

Table A.1. Recommendations for General-Purpose AI Models with Systemic Risk Providers, High-Risk System Providers, and High-Risk System Deployers

Recommendation General-Purpose AI Models with Systemic Risk (GPAISR) Providers High-Risk System Providers High-Risk System Deployers
Checkpoint system GPAISR providers should build foundational checkpoint infrastructure into their APIs, enabling automated pauses based on key risk indicators like rapid API calls, access attempts to sensitive resources, or suspicious outputs. This infrastructure should include default configurations, logging capabilities, and emergency shutdown options.
  • Measure II.6.1(7) of the Code of Practice: Agentic infrastructure
 High-risk system providers should then customize this infrastructure for their specific use cases by setting appropriate thresholds, defining clear human oversight protocols (including who can review or approve actions), and adding domain-specific checkpoints.
  • Art. 14 of the EU AI Act: Human oversight
 High-risk system deployers should ensure their staff have the requisite AI literacy, comprising such elements as technical knowledge, experience, and training to make informed and less biased decisions when reviewing actions at agentic checkpoints.
  • Art. 4 of the EU AI Act: AI literacy
Permission management system GPAISR providers should develop permission management systems for agent applications that enable granular control over agent capabilities and resource access. Providers should develop clear documentation regarding types of permissions and their relevant risks to support informed downstream deployment decisions.
  • Recital 101 and Annex XII(1)(d) of the EU AI Act: Transparency
  • Measure II.6.1(7) of the Code of Practice: Agentic infrastructure
  • Commitment I.1 of the Code of Practice: Documentation
 High-risk system providers should implement the permission management systems provided by model providers, configuring them to align with their specific operational requirements and risk profile.
  • Art. 14 of the EU AI Act: Human oversight
 High-risk system deployers should review and provide feedback on permission configurations based on operational experience.
  • Art. 26(5) of the EU AI Act: Operation-monitoring
  • Art. 27(1)(e): Fundamental rights impact assessment: Human oversight implementation

SOURCE: Table text drawn from Oueslati and Staes-Polet, Ahead of the Curve, which draws material from European Union, Artificial Intelligence Act; and European Commission, The General-Purpose AI Code of Practice.

Commercial Off-the-Shelf Solutions and Simple Recommendations

  • Google Cloud: Vertex AI and Cloud TPU⁠5
  • Google Development Kit: Loop Agents and Workflow Agents⁠6
  • HumanLoop⁠7
  • LangChain⁠8
  • “Deep Dive” (LinkedIn)⁠9

How Does This Relate to the Rest of the Guide or Other Threats That the User Cares About?

  • Autonomous: A.DSAP-2, II.NSS-2
  • Restrict access: A.ART-1, A.NSI-1, CC.ACA-1, CC.DPCC-1, CC.IRO-1, CC.NSI-1, COP.ACA-2, COP.ACA-3, COP.IRA-1, COP.IRA-2, COP.NSI-1, D.ACA-1, D.CSM-1, D.DPGC-2, D.DSSAC-1, D.DTS-1, D.NSI-1, HT.ACA-1, HT.HOAC-1, HT.HOAC-2, HT.PSAM-1, HT.PSAM-2, HT.RBA-1, HT.RBA-2, IE.ACA-2, IE.DPPC-1, IE.ISCM-1, IE.IV-1, IE.NSAP-1, IE.NSAP-2, IE.RMRL-1, IE.RMRL-2, IE.SML-1, IE.TD-1, IE.TD-2, II.ACA-1, II.AD-1, II.DPPC-1, II.DPPC-2, II.ISCM-1, II.ISCM-2, II.IVS-1, II.NSS-1, II.NSS-2, II.RMRL-1, II.RMRL-2, II.SML-1, MT.ACG-2, MW.ACA-1, MW.ACA-2, MW.NSC-1, MW.PSC-1
  • OWASP:
    • API4:2023—Unrestricted Resource Consumption
    • API6:2023—Unrestricted Access to Sensitive Business Flows

Other Sources of Information About This Topic

  • “Data Science and Engineering with Human in the Loop, Behind the Loop, and Above the Loop” (Harvard Data Science Review)⁠10
  • Artificial Intelligence Act (European Union)⁠11
  • “What Is Human in the Loop Machine Learning” (Medium)⁠12
  • “Pros and Cons of Autonomous Weapons Systems” (Military Review)⁠13
  • “From Assistant to Agent” (Credo.AI)⁠14
  • “Building Autonomous Systems” (Digital Ocean)⁠15
  • “Interrupts” (LangChain)⁠16
  • “AI Agent Security” (Medium)⁠17
  • “AI with Human Oversight” (Focalx)⁠18
  • Ahead of the Curve (The Future Society)⁠19

Notes

  1. Irwin and Greshake, “How Code Execution Drives Key Risks in Agentic AI Systems.” Return to content ⤴
  2. Attila, “AI Agent Tool Use.” Return to content ⤴
  3. OWASP, “Securing Agentic Applications Guide.” Return to content ⤴
  4. Oueslati and Staes‑Polet, “Ahead of the Curve.” Return to content ⤴
  5. Google Cloud, “What Is Human‑in‑the‑Loop (HITL) in AI & ML?.” Return to content ⤴
  6. Agent Development Kit, “Loop Agents” and “Workflow Agents.” Return to content ⤴
  7. Humanloop, “Agents.” Return to content ⤴
  8. LangChain, “LangChain Reference.” Return to content ⤴
  9. Sancheti, “Deep Dive.” Return to content ⤴
  10. Meng, “Data Science and Engineering with Human in the Loop, Behind the Loop, and Above the Loop.” Return to content ⤴
  11. European Union, “Artificial Intelligence Act, Article 14.” Return to content ⤴
  12. Bisen, “What Is Human in the Loop Machine Learning.” Return to content ⤴
  13. Etzioni and Etzioni, “Pros and Cons of Autonomous Weapons Systems.” Return to content ⤴
  14. Sherman et al., “From Assistant to Agent.” Return to content ⤴
  15. Payong and Mukherjee, “Building Autonomous Systems.” Return to content ⤴
  16. LangChain, “Interrupts.” Return to content ⤴
  17. Fessi, “AI Agent Security — Why You Should Pay Attention.” Return to content ⤴
  18. Agdestein, “AI with Human Oversight.” Return to content ⤴
  19. Oueslati and Staes‑Polet, “Ahead of the Curve.” Return to content ⤴