Data Manipulation
Tampering with Training Data to Influence Model Behavior
Description
Data manipulation attacks target the data used to train or adapt an AI system, affecting models across modalities, including language models, vision systems, speech recognition models, and biological or scientific ML pipelines. These attacks occur before or during training (e.g., during data collection, preprocessing, labeling, storage, or fine-tuning). The two types of attacks we focus on are as follows:
- Poisoning attacks inject malicious or mislabeled data samples so that the model learns incorrect or attacker-desired behavior.
- Backdoor attacks embed hidden “triggers” (e.g., a phrase, pixel pattern, or syntax) that cause misbehavior only when activated, making them harder to detect in predeployment testing.
Difficulty Level — Moderate-High
Data manipulation attacks require access to or influence over the training pipeline or one of its upstream data sources. The difficulty of data manipulation attacks is correlated with the amount of data required to conduct the attack:
- Poisoning attacks usually need thousands of altered samples but are easier to detect during validation and performance testing.
- Backdoor attacks need fewer data but are more difficult to detect.
Attack Consequence — High
Because AI models directly inherit properties from the data they learn from, corrupted data can cause
- incorrect system behavior in deployment
- embedded vulnerabilities that persist across retraining cycles
- significant operational and reputational damage
- regulatory or legal exposure when models trained on sensitive or scientific data are compromised
- high remediation costs, including data audits, retraining, reevaluation, and system downtime.
These risks apply equally to text systems, vision models, scientific models, and any ML system whose behavior is shaped by data. On the other hand, foundation models often train on massive, web-scraped datasets. An attacker might subtly pollute a public data source that is later ingested or directly tamper with stored data to poison the model. Backdoors can be implanted, remaining undetected during validation and enabling malicious behavior in deployed environments.