Securing the AI Lifecycle: Develop

The develop phase focuses on training and evaluation of an AI model to prepare it for deployment. This includes pretraining, fine-tuning, and evaluation, in which the model learns from large datasets, builds internal representations, and is assessed for safety, security, and performance before it is prepared for deployment. Because vulnerabilities introduced here can propagate into production environments, this phase requires rigorous governance, secure engineering practices, and tightly controlled infrastructure.

Pretraining and Instruction/Fine-Tuning

Training involves setting up and managing complex environments that use frameworks, such as PyTorch, and custom package managers, such as Conda. Poor dependency management, unsecured storage, or weak environment configurations can expose models to attack. Robust security controls and documentation are essential to protecting the development pipeline.

Subcategory	Identifier	Action or Control	OWASP and BIML Threats Mitigated	Why the Control Is Relevant
Privacy‑preserving techniques	MT.PPT-1	Embed anonymization, pseudonymization, and data minimization in model training processes	LLM03: Training Data Poisoning ML02: Data Poisoning Attack BIML: (training) Data Extraction	Embedding anonymization, pseudonymization, and data minimization in training processes helps protect sensitive information and reduces the likelihood of data poisoning, membership inference, or data extraction attacks.
Training environment security	MT.TES-2	Segment and isolate model training environments to ensure boundary protection	LLM05: Supply Chain Vulnerabilities API8:2023 — Security Misconfiguration ML05: Model Stealing Attack	Segmenting and isolating model training environments helps reduce unauthorized access and limits the potential impact of compromised components on the training process.
Access control and governance	MT.ACG-2	Enforce RBAC and least privilege to restrict access to LLM repositories and training environments	LLM03: Training Data Poisoning ML02: Data Poisoning Attack API5:2023 — Broken Function Level Authorization	Enforcing RBAC and least privilege for LLM repositories and training environments helps reduce the likelihood of unauthorized access, theft, or tampering with training data or models.
Adversarial robustness testing	MT.ART-1	Conduct red‑team campaigns using adversarial techniques to test and improve model robustness	ML01: Input Manipulation Attack ML02: Data Poisoning Attack LLM04: Model Denial of Service	Conducting red‑team campaigns using adversarial techniques helps identify and mitigate vulnerabilities by simulating realistic attacks, which can improve model resilience to malicious inputs and data perturbations.
Training process monitoring and validation	MT.TPMV-1	Monitor training loss and detect anomalies to identify potential model poisoning	LLM03: Training Data Poisoning ML02: Data Poisoning Attack BIML: Data Manipulation	Monitoring training loss and detecting anomalies help identify unusual behavior that may signal data poisoning or embedded backdoors, reducing the risk of compromised training processes.

Evaluation

Evaluation tests the model for accuracy, reliability, and security before deployment. Without secure evaluation processes, models may be exposed to tampering, biased performance metrics, or overlooked vulnerabilities. Using controlled environments, strict access management, and documented testing protocols ensures accurate and secure evaluations.

Subcategory	Identifier	Action or Control	OWASP and BIML Threats Mitigated	Why the Control Is Relevant
Adversarial robustness testing	E.ART-1	Simulate attacks to identify security weaknesses in the model and systems	ML01: Input Manipulation Attack LLM01: Prompt Injection	Simulating attacks helps identify weaknesses in how the model responds to crafted inputs, reducing the likelihood of successful manipulation or poisoning attempts.
Threat evaluation	E.TE-2	Conduct fuzzing and penetration testing to validate model and environment security	LLM01: Prompt Injection ML01: Input Manipulation Attack API8:2023 — Security Misconfiguration	Conducting fuzzing and penetration testing helps uncover vulnerabilities in the model and its environment by exposing them to malicious or unexpected inputs, which reduces risks associated with prompt injection, denial‑of‑service conditions, input manipulation, and misconfigurations.
Data and code integrity validation	E.DCIV-1	Verify model code and test datasets for tampering during testing	LLM03: Training Data Poisoning ML02: Data Poisoning Attack	Verifying the integrity of model code and test datasets helps detect tampering or unauthorized modifications, reducing the likelihood of introducing vulnerabilities, biases, or malicious behaviors during evaluation.
Continuous security monitoring	E.CSM-1	Implement continuous security monitoring and protection for all systems and networks	API8:2023 — Security Misconfiguration LLM04: Model Denial of Service ML06: AI Supply Chain Attack	Continuous security monitoring helps detect and respond to misconfigurations, service disruptions, or compromised components, reducing risks to availability, data integrity, and the supply chain.
Infrastructure security and configuration management	E.ISCM-1	Implement and maintain security measures for infrastructure, including data protection and monitoring	API8:2023 — Security Misconfiguration LLM05: Supply Chain Vulnerabilities ML06: AI Supply Chain Attack	Implementing and maintaining infrastructure security measures supports proper configuration and monitoring, reducing risks tied to misconfigurations, inaccurate inventory, or supply chain weaknesses.
Adversarial robustness testing	E.ART-2	Test the model with adversarial examples to check for vulnerabilities	ML01: Input Manipulation Attack ML08: Model Skewing BIML: Input Manipulation	Testing models with adversarial examples helps identify weaknesses in how they handle manipulated inputs, reducing the likelihood that crafted inputs could trigger incorrect or skewed predictions.

The decisions made during the model development phase play a crucial role in determining the performance and reliability of the final AI model. Once trained, the model will be released to the public, facing an almost infinite space of potential inputs, which can result in unpredictable outputs. To safeguard the company's intellectual property and ensure user safety, it is imperative that the model undergo comprehensive training and rigorous testing. This process not only optimizes the model's performance but also helps identify and mitigate potential risks associated with its deployment in real-world scenarios. By prioritizing thorough evaluation, organizations can enhance the robustness of their AI applications and protect both their assets and their users.