Develop

Table B.6. Security Controls Related to Model Training

Subcategory	Identifier	Action or Control	OWASP and BIML Threats Mitigated	Why the Control Is Relevant
Access control and governance	MT.ACG-1	Follow secure design and coding practices during model training.	LLM03: Training Data Poisoning ML02: Data Poisoning Attack ML07: Transfer Learning Attack ML10: Model Poisoning LLM05: Supply Chain Vulnerabilities ML06: AI Supply Chain Attack	Ensures that secure design and coding practices are followed to prevent manipulation or poisoning of training data, embedding of backdoors, and supply chain vulnerabilities
Access control and governance	MT.ACG-2	Enforce RBAC and least privilege to restrict access to LLM repositories and training environments.	API5:2023—Broken Function Level Authorization LLM03: Training Data Poisoning LLM10: Model Theft ML02: Data Poisoning Attack ML05: Model Stealing Attack	Prevents unauthorized users from accessing, stealing, or tampering with models and training data
Adversarial robustness testing	MT.ART-1	Conduct red-team campaigns using adversarial techniques to test and improve model robustness.	LLM04: Model Denial of Service ML01: Input Manipulation Attack ML02: Data Poisoning Attack ML08: Model Skewing	Helps identify and mitigate vulnerabilities by simulating attacks, ensuring that the model can withstand data perturbations and malicious inputs
Adversarial robustness testing	MT.ART-2	Train the model with adversarial inputs to enhance robustness.	LLM01: Prompt Injection ML01: Input Manipulation Attack ML08: Model Skewing	Enhances model robustness by training the model to recognize and correctly handle adversarial inputs, reducing the likelihood of successful manipulation
Data quality and integrity assurance	MT.DQIA-1	Implement measures to prevent poisoned data from affecting model weights during training.	LLM03: Training Data Poisoning ML02: Data Poisoning Attack BIML: Data Manipulation	Ensures that the training data are verified and cleaned to prevent malicious data from altering the model’s weights and behavior
Data quality and integrity assurance	MT.DQIA-2	Validate the model with an unused validation set to detect data poisoning.	LLM03: Training Data Poisoning ML02: Data Poisoning Attack BIML: Data Manipulation	Helps identify discrepancies and anomalies that may indicate data poisoning, ensuring the integrity of the training process
Infrastructure security management	MT.ISM-1	Implement and maintain security measures for infrastructure, including data protection and monitoring.	API8:2023—Security Misconfiguration API9:2023—Improper Inventory Management LLM05: Supply Chain Vulnerabilities ML06: AI Supply Chain Attack	Ensures proper configuration and monitoring of infrastructure to prevent misconfigurations, maintain accurate inventory, and secure supply chain components
Infrastructure security management	MT.ISM-2	Regularly scan and patch vulnerabilities in model training environments.	API8:2023—Security Misconfiguration LLM05: Supply Chain Vulnerabilities ML06: AI Supply Chain Attack	Ensures that any weaknesses in the model training environment are identified and fixed, preventing exploitation through compromised components or misconfigurations
Model protection and anti-theft	MT.MPAT-1	Add a watermark to the model’s code and training data to trace theft sources.	LLM10: Model Theft ML05: Model Stealing Attack ML06: AI Supply Chain Attack Model Extraction	Helps trace the source of theft, making it easier to identify and hold attackers accountable, thereby deterring unauthorized access and replication
Model protection and anti-theft	MT.MPAT-2	Pause training if the pretraining model’s capabilities match or exceed the given model’s capabilities.	LLM05: Supply Chain Vulnerabilities LLM10: Model Theft ML05: Model Stealing Attack ML06: AI Supply Chain Attack	Ensures that pretraining models do not introduce vulnerabilities or exceed security standards before continuing training, preventing unauthorized access or replication
Privacy-preserving techniques	MT.PPT-1	Embed anonymization, pseudonymization, and data minimization in model training processes.	LLM03: Training Data Poisoning LLM06: Sensitive Information Disclosure ML02: Data Poisoning Attack ML04: Membership Inference Attack ML10: Model Poisoning BIML: Data Manipulation BIML: (training) Data Extraction	Ensures that sensitive information is protected and reduces the risk of data poisoning, membership inference, and data extraction by anonymizing and minimizing the data used during training
	MT.PPT-2	Apply friendly noise to training examples to mitigate the effects of poisoned data.	LLM03: Training Data Poisoning ML02: Data Poisoning Attack BIML: Data Manipulation	Helps mitigate the impact of poisoned data by making it harder for attackers to introduce specific vulnerabilities or biases without degrading model performance
	MT.PPT-3	Apply L1 or L2 regularization to prevent model overfitting and improve generalization.	ML04: Membership Inference Attack ML10: Model Poisoning	Prevents overfitting, reducing the model’s ability to memorize training data and thus mitigating membership inference attacks and model skewing
Supply chain and package security	MT.SCPS-1	Sign Git commits with GNU Privacy Guard (GPG) keys and load only trusted serialized ML models.	API10:2023—Unsafe Consumption of APIs LLM05: Supply Chain Vulnerabilities ML05: Model Stealing Attack ML06: AI Supply Chain Attack	Ensures the integrity and authenticity of the source code and models by verifying signatures and loading models from trusted sources, preventing compromised components from being introduced into the pipeline
Supply chain and package security	MT.SCPS-2	Update and review secure packages before using them in virtual environments for development.	API10:2023—Unsafe Consumption of APIs LLM05: Supply Chain Vulnerabilities ML06: AI Supply Chain Attack	Ensures that only secure and reviewed packages are used, preventing compromised components from introducing vulnerabilities into the model
Training environment security	MT.TES-1	Harden and securely configure all training environments.	API8:2023—Security Misconfiguration LLM05: Supply Chain Vulnerabilities ML02: Data Poisoning Attack ML06: AI Supply Chain Attack ML10: Model Poisoning	Prevents misconfigurations, protects against compromised components, and mitigates risks of data poisoning and model skewing
Training environment security	MT.TES-2	Segment and isolate model training environments to ensure boundary protection.	API8:2023—Security Misconfiguration LLM05: Supply Chain Vulnerabilities LLM10: Model Theft ML05: Model Stealing Attack ML06: AI Supply Chain Attack	Prevents unauthorized access and reduces the risk of compromised components affecting the training process
Training process monitoring and validation	MT.TPMV-1	Monitor training loss and detect anomalies to identify potential model poisoning.	LLM03: Training Data Poisoning ML02: Data Poisoning Attack ML07: Transfer Learning Attack BIML: Data Manipulation	Helps identify and mitigate attempts to poison the training data or embed backdoors by flagging unusual model behavior during training
	MT.TPMV-2	Regularly evaluate model performance to detect degradation from drift or data changes.	LLM03: Training Data Poisoning ML02: Data Poisoning Attack ML10: Model Poisoning	Helps identify and mitigate issues caused by model skewing, data poisoning, and training data poisoning by detecting unexpected changes or degradation in model behavior
	MT.TPMV-3	Use statistical and ML methods to detect and alert on feedback data anomalies.	LLM03: Training Data Poisoning ML02: Data Poisoning Attack ML10: Model Poisoning	Helps identify and mitigate attempts to corrupt training data or manipulate model behavior

Table B.7. Security Controls Related to Evaluation

Subcategory	Identifier	Action or Control	OWASP and BIML Threats Mitigated	Why the Control Is Relevant
Adversarial robustness testing	E.ART-1	Simulate attacks to identify security weaknesses in the model and systems.	LLM01: Prompt Injection ML01: Input Manipulation Attack ML02: Data Poisoning Attack ML07: Transfer Learning Attack ML08: Model Skewing	Helps identify vulnerabilities in the model’s response to crafted inputs, ensuring robustness against manipulation and poisoning
Adversarial robustness testing	E.ART-2	Test the model with adversarial examples to check for vulnerabilities.	ML01: Input Manipulation Attack ML08: Model Skewing BIML: Input Manipulation	Helps identify and mitigate vulnerabilities in which crafted inputs could deceive the model into making incorrect predictions
Bias and fairness evaluation	E.BFE-1	Test and mitigate biases in model predictions to ensure performance and fairness.	LLM03: Training Data Poisoning ML02: Data Poisoning Attack ML10: Model Poisoning	Ensures that biases introduced during training are identified and mitigated, preventing degradation of model performance and fairness
Continuous security monitoring	E.CSM-1	Implement continuous security monitoring and protection for all systems and networks.	API8:2023—Security Misconfiguration API9:2023—Improper Inventory Management LLM04: Model Denial of Service ML06: AI Supply Chain Attack	Helps detect and respond to misconfigurations, manage inventory, prevent denial-of-service attacks, and identify compromised components in the AI supply chain
Data and code integrity validation	E.DCIV-1	Verify model code and test datasets for tampering during testing.	LLM03: Training Data Poisoning ML02: Data Poisoning Attack ML07: Transfer Learning Attack ML10: Model Poisoning BIML: Data Manipulation	Ensures that the model and test datasets have not been altered to introduce vulnerabilities, biases, or malicious behaviors during the evaluation phase
Infrastructure security and configuration management	E.ISCM-1	Implement and maintain security measures for infrastructure, including data protection and monitoring.	API8:2023—Security Misconfiguration API9:2023—Improper Inventory Management LLM05: Supply Chain Vulnerabilities ML06: AI Supply Chain Attack	Ensures proper configuration and monitoring of infrastructure to prevent misconfigurations, maintain accurate inventory, and secure supply chain components
Network security controls	E.NSC-1	Implement diverse system components to minimize the risk of widespread compromise.	API8:2023—Security Misconfiguration LLM05: Supply Chain Vulnerabilities ML06: AI Supply Chain Attack	Helps isolate and contain potential compromises, reducing the risk of a single point of failure and mitigating the impact of supply chain vulnerabilities and misconfigurations
Quality assurance	E.QA-1	Record, test, and approve all changes.	API8:2023—Security Misconfiguration LLM05: Supply Chain Vulnerabilities ML06: AI Supply Chain Attack	Ensures that all changes are properly documented, tested, and approved, reducing the risk of misconfigurations and vulnerabilities introduced through unverified changes
Security assessment	E.SA-1	Perform security testing to identify and mitigate AI model vulnerabilities before deployment.	LLM01: Prompt Injection LLM02: Insecure Output Handling LLM03: Training Data Poisoning LLM04: Model Denial of Service LLM06: Sensitive Information Disclosure LLM07: Insecure Plugin Design LLM10: Model Theft ML01: Input Manipulation Attack ML02: Data Poisoning Attack ML03: Model Inversion Attack ML04: Membership Inference Attack ML05: Model Stealing Attack ML07: Transfer Learning Attack ML08: Model Skewing ML09: Output Integrity Attack ML10: Model Poisoning	Ensures that vulnerabilities in the AI model are identified and mitigated before deployment, preventing various attacks and unauthorized access
Security assessment	E.SA-2	Perform comprehensive security testing to identify vulnerabilities and weaknesses in the entire system.	API2:2023—Broken Authentication API3:2023—Broken Object Property Level Authorization API5:2023—Broken Function Level Authorization API8:2023—Security Misconfiguration LLM01: Prompt Injection LLM02: Insecure Output Handling LLM03: Training Data Poisoning ML01: Input Manipulation Attack ML02: Data Poisoning Attack ML07: Transfer Learning Attack	Helps identify and mitigate vulnerabilities, such as broken authentication, authorization flaws, security misconfigurations, and various injection and poisoning attacks, by thoroughly evaluating the system’s source code and configurations
Threat evaluation	E.TE-1	Implement a framework to evaluate novel threats from advanced AI with dangerous capabilities.	LLM01: Prompt Injection LLM03: Training Data Poisoning LLM04: Model Denial of Service LLM06: Sensitive Information Disclosure LLM08: Excessive Agency ML01: Input Manipulation Attack ML02: Data Poisoning Attack ML07: Transfer Learning Attack ML08: Model Skewing	Ensures that novel threats from advanced AI capabilities are identified and mitigated, preventing manipulation, deception, cyber-offense, and other dangerous actions
Threat evaluation	E.TE-2	Conduct fuzzing and penetration testing to validate model and environment security.	API8:2023—Security Misconfiguration LLM01: Prompt Injection LLM04: Model Denial of Service ML01: Input Manipulation Attack ML08: Model Skewing	Helps identify vulnerabilities in the model and environment by simulating malicious inputs and attacks, ensuring robustness against prompt injections, denial of service, input manipulation, adversarial examples, and misconfigurations