Input Extraction

Inferring Sensitive Input Data from Model Outputs

Description

Input extraction attacks, also known as model inversion attacks, aim to reconstruct or infer sensitive user inputs by analyzing a model’s outputs. Unlike data extraction attacks, input extraction focuses on information that users provide during inference, such as text queries, uploaded images, audio samples, sensor data, and biological results. Attackers can infer or approximate sensitive attributes by exploiting

correlations learned during training
overconfident outputs or probability distributions
embedding representations that expose sensitive attributes
patterns in generative outputs that leak details about a user’s query or uploaded content.

Attacker Motivation

These attacks may be used for surveillance, data theft, or impersonation. They may also target a model’s business value by reverse engineering its decisionmaking or proprietary data patterns, which can enable cloning for competitive advantage or bypassing built-in security features that enable the attacker to misuse the model.

Demonstrated Input Extraction Attacks

In 2015, researchers demonstrated early input extraction techniques and showed that attackers could reconstruct recognizable faces or infer sensitive genetic attributes from model outputs.

More recent studies in 2023 and 2024 from Princeton, Google DeepMind, and the University of Maryland demonstrated that modern generative models can unintentionally reveal training images or allow attackers to reconstruct private inputs through inversion and repeated sampling.

Implication 1: Attacks do not rely on detailed outputs—they can extract or reconstruct using only sampled outputs.
Implication 2: Attacks can extract data not just from an input but from the model’s distribution itself, enabling the attacker to reconstruct anything the model saw.

SOURCES: Fredrikson, Jha, and Ristenpart, “Model Inversion Attacks That Exploit Confidence Information and Basic Countermeasures”; Carlini et al., “Extracting Training Data from Diffusion Models"; Marchal et al., “Generative AI Misuse.”

Difficulty Level — Moderate-High

Can be performed as black-box attacks (no access to model internals) or white-box attacks (full access to model internals).

Attack Consequence — Moderate-High

Risks include privacy violations involving regulated data (e.g., Health Insurance Portability and Accountability Act [HIPAA]-protected health information), exposure of individuals’ participation in a dataset without their consent or sufficient anonymization, loss of competitive advantage, reputational damage, and higher costs to strengthen anomaly detection and defenses.

Next: Data Extraction