Securing the AI Lifecycle: Design

The AI lifecycle starts with the design phase and consists of four foundational elements: algorithms, data, human talent, and compute. These components form the foundational blueprint of many AI systems and must be secured from the initial conception.

Algorithms

Algorithms include the core design decisions that define what the AI system does and how it functions. Protecting algorithms and algorithmic design—including model architecture, preprocessing steps, engineering, and control logic—helps prevent reverse engineering, intellectual property theft, and targeted attacks that could manipulate outputs or expose sensitive information.

Subcategory	Identifier	Action or Control	OWASP and BIML Threats Mitigated	Why the Control Is Relevant
Adversarial robustness testing	A.ART-1	Perform regular penetration tests and breach simulations to evaluate trust boundaries and access controls	API1:2023 — Broken Object Level Authorization LLM01: Prompt Injection ML01: Input Manipulation Attack	Regular penetration tests and breach simulations help identify weaknesses in trust boundaries and access controls, reducing the likelihood that adversarial inputs or unauthorized access attempts will succeed.
Data storage and access protection	A.DSAP-1	Encrypt data and implement access controls to prevent unauthorized access and theft	API1:2023 — Broken Object Level Authorization LLM10: Model Theft ML05: Model Stealing Attack ML06: AI Supply Chain Attack BIML: Model Extraction	Encrypting data and implementing access controls help reduce the risk of unauthorized access, theft, or manipulation of sensitive data and models.
Privacy-preserving techniques	A.PPT-1	Implement differential privacy to protect individual records and prevent malicious knowledge transfer	LLM03: Training Data Poisoning ML02: Data Poisoning Attack BIML: Input Extraction	Applying differential privacy helps protect individual records in training data, reducing the likelihood of sensitive information inference or malicious data injection.
Cryptographic protection and confidential computing	A.CPCC-1	Use cryptographic keys to create a secure enclave for data during processing	LLM06: Sensitive Information Disclosure ML03: Model Inversion Attack BIML: Input Extraction	Confidential computing helps ensure that data and model processing occur within secure enclaves, reducing the risk of unauthorized access or sensitive information extraction.
Data integrity and verification	A.DIV-1	Verify integrity of training datasets and model code using hashing, checksums, and digital signatures	LLM03: Training Data Poisoning ML02: Data Poisoning Attack BIML: Model Manipulation	Verifying dataset and code integrity using hashing and signing techniques helps detect tampering and reduces the likelihood of unauthorized modifications within the AI pipeline.

Human Talent

Human talent involves the expertise and management of the people building and maintaining the system. Staff—including developers, data scientists, ML engineers, and domain experts—are critical to quality and security but can be targets of social engineering or insider threats. Safeguarding human resources protects intellectual property and helps reduce risk.

Subcategory	Identifier	Action or Control	OWASP and BIML Threats Mitigated	Why the Control Is Relevant
Physical security and access management	HT.PSAM-1	Restrict access to sensitive data based on security clearance levels	API1:2023 — Broken Object Level Authorization LLM06: Sensitive Information Disclosure ML03: Model Inversion Attack ML06: AI Supply Chain Attack	Limiting access to sensitive data based on security clearance levels helps reduce the likelihood of unauthorized access or misuse by ensuring that only appropriately cleared personnel can view or handle the information.
Compliance and regulatory training	HT.CRT-1	Implement access regulation, security awareness training, and role-based security training per applicable security standards	API1:2023 — Broken Object Level Authorization LLM07: Insecure Plugin Design ML06: AI Supply Chain Attack	Providing proper access control, authorization, and security configurations through training and compliance with established security standards (CMMC 2.0, NIST SP 800‑171, TS//SCI) reduces risk of misconfigurations or supply chain weaknesses.
Security training and awareness programs	HT.STAP-1	Train employees to identify and respond to social engineering attacks	LLM01: Prompt Injection ML01: Input Manipulation Attack	Training employees to identify and respond to social engineering attempts helps reduce the risk of input manipulation and unauthorized data access, supporting defenses against prompt injection, malicious inputs, and data extraction attacks.
Access control and authorization	HT.ACA-1	Grant access to sensitive data only as needed, based on least privilege	API1:2023 — Broken Object Level Authorization LLM06: Sensitive Information Disclosure ML03: Model Inversion Attack	Applying least‑privilege access controls helps reduce the likelihood of unauthorized data access or leakage by limiting sensitive information to those personnel with a legitimate need.
Security training and awareness programs	HT.STAP-2	Conduct regular security awareness, education, and training sessions for all employees	API8:2023 — Security Misconfiguration LLM05: Supply Chain Vulnerabilities ML06: AI Supply Chain Attack	Conducting regular security awareness and education sessions helps reduce the risk of misconfigurations, supply chain vulnerabilities, overreliance on AI systems, and unintended disclosure of sensitive information by keeping employees informed of security best practices.

Compute

Compute is the infrastructure and computational resources necessary for training and deploying AI models. Improper controls can expose systems to attacks, such as distributed denial-of-service (DDoS) attacks, cloud exploits, or hijacked compute. Secure access and continuous monitoring are essential for secure, reliable operations.

Subcategory	Identifier	Action or Control	OWASP and BIML Threats Mitigated	Why the Control Is Relevant
Network security and isolation	COP.NSI-1	Deploy a firewall to scan and validate all requests and responses for attack signatures	API1:2023 — Broken Object Level Authorization LLM01: Prompt Injection ML01: Input Manipulation Attack	Using a firewall to scan and validate requests and responses helps detect and block attack signatures, reducing the likelihood of unauthorized access, resource abuse, or malicious inputs.
Access control and authorization	COP.ACA-3	Limit entry to areas with sensitive data to authorized personnel only	ML01: Input Manipulation Attack ML02: Data Poisoning Attack ML03: Model Inversion Attack	Restricting physical access to sensitive areas helps reduce the risk of unauthorized personnel accessing data, models, or systems, thereby lowering exposure to theft, tampering, or extraction.
Data protection and confidential computing	COP.DPCC-1	Implement confidential computing for all applicable network services and information transfers	ML03: Model Inversion Attack BIML: (training) Data Extraction BIML: Input Extraction	Implementing confidential computing helps protect data and computations during processing, reducing the risk of sensitive information disclosure, unauthorized extraction, or manipulation.
	COP.DPCC-2	Store sensitive data in secure environments with controlled access	BIML: (training) Data Extraction ML03: Model Inversion Attack ML04: Membership Inference Attack	Storing sensitive data in secure, access‑controlled environments helps decrease the likelihood of unauthorized access or extraction, supporting confidentiality and reducing manipulation risks.
Access control and authorization	COP.ACA-1	Harden training systems and environments to reduce attack surfaces	LLM03: Training Data Poisoning LLM05: Supply Chain Vulnerabilities ML02: Data Poisoning Attack	Hardening training systems and environments helps reduce vulnerabilities that could be exploited for data poisoning, supply chain compromise, or insertion of backdoors.

Subcategory	Identifier	Action or Control	OWASP and BIML Threats Mitigated	Why the Control Is Relevant
Data protection and confidential computing	CC.DPCC-1	Implement confidential computing for all applicable network services and information transfers	API6:2023 — Unrestricted Access to Sensitive Business Flows LLM06: Sensitive Information Disclosure ML03: Model Inversion Attack BIML: Input Extraction	Implementing confidential computing helps protect data and computations from unauthorized access or tampering, reducing the likelihood of sensitive information disclosure, model extraction, or inference attacks.
	CC.DPCC-2	Store sensitive data in secure cloud environments with controlled access	LLM06: Sensitive Information Disclosure ML03: Model Inversion Attack API8:2023 — Security Misconfiguration	Storing sensitive data in secure cloud environments with controlled access helps reduce the risk of unauthorized access or manipulation, supporting the protection of sensitive information and lowering exposure to data extraction or model skewing.
Network security and isolation	CC.NSI-1	Implement firewalls, IDS/IPS, and network segmentation in cloud‑based training environments	API4:2023 — Unrestricted Resource Consumption LLM04: Model Denial of Service ML06: AI Supply Chain Attack	Using firewalls, IDS/IPS, and network segmentation in cloud‑based training environments helps reduce unauthorized access, limit malicious traffic, and support proper configuration to mitigate risks such as resource exhaustion, sensitive flow exposure, and supply chain attacks.
Access control	CC.AC-1	Implement RBAC to restrict cloud data access based on user roles	API1:2023 — Broken Object Level Authorization LLM07: Insecure Plugin Design	Implementing RBAC helps ensure that only authorized users can access specific data and functionalities based on their roles, reducing the likelihood of unauthorized access or misuse.
Training environment security	CC.TES-1	Harden training systems and environments to reduce attack surfaces	LLM03: Training Data Poisoning ML02: Data Poisoning Attack API8:2023 — Security Misconfiguration LLM05: Supply Chain Vulnerabilities	Hardening training systems and environments helps reduce attack surfaces and lowers the risk of unauthorized access or manipulation, supporting the integrity and security of training data and components.

Data

Data refers to the collection and preparation of data for training and testing AI models. Quality and security of data are critical to the performance and reliability of the AI application. Poor handling can lead to data poisoning, unauthorized access, or compliance risks. Strong governance practices and security controls ensure data integrity and reliability.

Subcategory	Identifier	Action or Control	OWASP and BIML Threats Mitigated	Why the Control Is Relevant
Continuous security monitoring	D.CSM-1	Continuously monitor systems for unauthorized access or anomalies involving sensitive data	API1:2023 — Broken Object Level Authorization LLM06: Sensitive Information Disclosure ML03: Model Inversion Attack	Continuous monitoring of systems for unauthorized access or anomalies helps detect and respond to suspicious activity, reducing the likelihood of unauthorized data access, sensitive information disclosure, or model manipulation.
Data storage security and access control	D.DSSAC-1	Encrypt stored data and implement access controls to protect information at rest	API6:2023 — Unrestricted Access to Sensitive Business Flows LLM06: Sensitive Information Disclosure ML03: Model Inversion Attack BIML: Model Extraction BIML: (training) Data Extraction	Encrypting stored data and applying access controls help reduce the risk of unauthorized access or extraction of sensitive information, supporting data integrity and confidentiality.
Data transmission security	D.DTS-1	Encrypt data in transit between AI models, APIs, and applications	API2:2023 — Broken Authentication LLM06: Sensitive Information Disclosure ML03: Model Inversion Attack BIML: (training) Data Extraction	Encrypting data in transit helps protect sensitive information from interception or unauthorized access, reducing the likelihood of disclosure or manipulation during transmission.
Access control and authorization	D.ACA-1	Restrict access to sensitive data based on security clearance levels	API1:2023 — Broken Object Level Authorization LLM06: Sensitive Information Disclosure ML03: Model Inversion Attack	Restricting access to sensitive data based on security clearance levels helps reduce the likelihood of unauthorized access or misuse by ensuring that only appropriately cleared personnel can access the information.
Cryptographic protection and encryption	D.CPE-1	Implement homomorphic encryption for secure data analysis and privacy‑preserving ML	LLM06: Sensitive Information Disclosure ML03: Model Inversion Attack BIML: (training) Data Extraction	Implementing homomorphic encryption helps maintain data confidentiality during analysis and ML processes, reducing the risk of unauthorized access or sensitive information extraction.