Misuse-Resilience Testing

Resilience testing is a method for evaluating how well a system can withstand and recover from failures, crashes, or unexpected disruptions. According to software testing firm Frugal Testing, this involves

  • validating fault tolerance, such as handling network outages, server crashes, or hardware malfunctions
  • maintaining core functionality, even when some components fail
  • simulating real-world incident scenarios
  • implementing automatic recovery mechanisms.1

When it comes to AI systems, cybersecurity firm Cyber Resilience highlights additional considerations. It recommends

  • defining clear measures for identity verification
  • mitigating credential-based attacks through two-factor authentication, such as YubiKeys, RSA SecurID tokens, or smart cards
  • conducting routine patching
  • providing personnel training to withstand social engineering.2

Another useful approach is canary releasing, which involves rolling out new versions of ML models to small subsets of users. This allows organizations to test models in real-world applications and identify potential risks before making them fully available.

For LLMs specifically, resilience testing can be accomplished by systematically assessing whether a model’s output remains consistent with intended product boundaries, even when the model is challenged with difficult or adversarial prompts. AI evaluation firm Galtea recommends the following steps:

  • defining the scope for what actions are allowed or prohibited
  • classifying input as in scope or out of scope
  • judging the model’s response
  • assigning a score that reflects the model’s ability to remain robust even under adversarial conditions.3

Commercial Off-the-Shelf Solutions and Simple Recommendations

How Does This Relate to the Rest of the Guide or Other Threats That the User Cares About?

  • Preventing misuse: CC.ACA-1, D.ACA-1, HT.ACA-3, HT.CIS-1, HT.HOAC-1, HT.PSAM-1, HT.PSAM-1, HT.RBA-2, IE.ACA-1, IE.DPPC-1, IE.IRSO-2, IE.SML-1, II.ACA-2, II.IRSO-1
  • Resilience: CC.IRO-1, COP.IRA-1, COP.IRA-2

Other Sources of Information About This Topic

  • “System Resilience Part 6” (Carnegie Mellon University)⁠15
  • “AI and Misuse” (Resilience)⁠16
  • “Resilience Testing” (Tricentis)⁠17
  • “What Is Resilience Testing?” (Frugal Testing)⁠18
  • “The Art of Software Survival” (Medium)⁠19
  • “Canary Releases and Blue-Green Deployments for ML Models” (Bugfree.ai)⁠20
  • “Model Deployment” (DagsHub)⁠21
  • “Shadow Deployment vs. Canary Release of Machine Learning Models” (JFrog ML)⁠22
  • “Dynamic A/B Testing for Machine Learning Models with Amazon SageMaker MLOps Projects” (AWS)⁠23

Notes

  1. Garg, “What Is Resilience Testing?” Return to content ⤴
  2. Saade, “AI and Misuse.” Return to content ⤴
  3. Galtea, “Misuse Resilience.” Return to content ⤴
  4. Tricentis, “Tricentis NeoLoad.” Return to content ⤴
  5. Netflix, “Chaos Monkey.” Return to content ⤴
  6. Gremlin, “Homepage.” Return to content ⤴
  7. LitmusChaos, “Homepage.” Return to content ⤴
  8. Lucas, “Resilience Testing with Toxiproxy.” Return to content ⤴
  9. Jepsen, “Homepage.” Return to content ⤴
  10. Sathaye and Chintalapati, “Getting Started with Istio on Amazon EKS.” Return to content ⤴
  11. Tetrate, “Envoy AI Gateway V0.2.” Return to content ⤴
  12. BrowserStack, “JMeter Stress Testing.” Return to content ⤴
  13. Gatling, “Homepage.” Return to content ⤴
  14. Galtea, “Misuse Resilience.” Return to content ⤴
  15. Firesmith, “System Resilience Part 6.” Return to content ⤴
  16. Saade, “AI and Misuse.” Return to content ⤴
  17. Tricentis, “Resilience Testing.” Return to content ⤴
  18. Garg, “What Is Resilience Testing?” Return to content ⤴
  19. Panicker, “The Art of Software Survival.” Return to content ⤴
  20. Bugfree.ai, “Canary Releases and Blue‑Green Deployments for ML Models.” Return to content ⤴
  21. Martin, “Model Deployment.” Return to content ⤴
  22. Mikulski, “Shadow Deployment vs. Canary Release of Machine Learning Models.” Return to content ⤴
  23. Bright, “Dynamic A/B Testing for Machine Learning Models with Amazon SageMaker MLOps Projects.” Return to content ⤴