AI Model Theft: Extraction and Privacy Attacks

AI model theft does not always mean breaking into a repository and downloading a weights file. A public or partner-facing API may reveal enough information for an adversary to train a substitute model on the target’s answers. Other attacks try to determine whether a particular record belonged to the training set, reconstruct features, or extract memorised content.

In one paragraph: model extraction, membership inference, model inversion, and training-data extraction are different objectives that require different tests and controls. Query limits help but provide no guarantee. Defence should combine response minimisation, customer identity and cost controls, detection of unusual query sequences, empirical privacy testing, artefact protection, and a clear decision about what actually constitutes valuable intellectual property.

NIST organises modern attacks on learning systems in Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations. The report covers privacy attacks, predictive and generative models, and limitations of available mitigations. It is a valuable baseline because marketing language often collapses every AI leak into one category.

Four different attacker questions

Class	Attacker’s question	Protected asset
model extraction	“Can I copy the model’s function?”	intellectual property and decision logic
membership inference	“Was record X in training?”	individual or organisational privacy
model inversion	“Which data features can I reconstruct?”	attributes and data representations
training-data extraction	“Can the model emit a memorised sample?”	exact training content

These classes may support one another, but they are not synonyms. A substitute model can make further analysis easier, yet high response agreement does not prove recovery of the original parameters. Membership inference can breach privacy without revealing a complete record. Reports and executive communication must state precisely what the test confirmed.

Model extraction: copying behaviour through an API

The USENIX Security 2016 paper Stealing Machine Learning Models via Prediction APIs demonstrated that black-box API access could be enough to reproduce functionally similar models across several popular model classes. Confidence scores and rich decision information were especially useful, but the authors also showed that hiding confidence alone did not eliminate extraction risk.

An adversary builds a query dataset, collects labels or scores, and trains a surrogate model. Their goal may be to:

• avoid paying for a commercial API;
• clone a scoring or moderation mechanism;
• test evasion offline at scale;
• learn decision boundaries of a security system;
• build a competing product that imitates the service.

For generative AI, “agreement” is more difficult to define than for a classifier. Sampling, system prompts, tools, retrieval, and service updates all change output. An adversary may still clone a specialised function, style, policy, or pipeline. This is why LLM evaluation needs several task-specific metrics rather than one text-similarity score.

Membership inference: privacy of one record

The pioneering IEEE Symposium on Security and Privacy 2017 paper Membership Inference Attacks Against Machine Learning Models examined whether model access can reveal that a given record belonged to the training dataset. Statistical differences in model behaviour for seen and unseen examples can create a membership signal.

The impact is contextual. Learning that a record belonged to a clinic’s patient set, a financial assistance programme, or a fraud-victim dataset can itself be sensitive. The attack does not have to reconstruct the full record to violate privacy.

Risk often increases with overfitting, detailed probability outputs, and strong auxiliary knowledge. This is not a simple equation in which “no overfitting means no leakage.” Empirical testing must reflect a realistic attacker and the deployed interface.

Inversion and training-data extraction

Model inversion aims to reconstruct features or representative information from model outputs. Training-data extraction seeks specific memorised content. In generative systems, rare, repeated, or unique strings are particularly relevant, but risk depends on the dataset, training process, and interface.

Not every similar fragment proves memorisation. A language model may generate a common sentence from the learned distribution or repeat content supplied by the current RAG context. A test must control context sources, model version, sampling parameters, and whether the document exists in the retrieval index. An AI security audit should distinguish leakage from weights from an authorization defect in RAG.

Threat-modelling a business AI API

Start with assets. Is the value in weights, training data, the system prompt, a safety policy, retrieval documents, embeddings, expert labels, or the complete workflow? A publicly available base model may not be secret, while domain fine-tuning and surrounding decision logic are.

Then define attacker capabilities:

• anonymous access or verified account;
• hard labels, confidence, log probabilities, embeddings, or token streaming;
• query volume and cost;
• ability to create multiple accounts;
• domain knowledge and access to a similar base model;
• observation of several service versions;
• batch or adaptive querying.

Finally, define success. “Model theft” without a metric is not testable. For a classifier, success may be agreement on an independent dataset; for a ranker, ordering correlation; for moderation, prediction of policy decisions; and for an LLM, performance on a private task suite.

Testing without exposing real data

1. Use a dedicated model or test tenant

Privacy testing against production records can create an incident. Prepare canary records, synthetic samples, and a model with a known dataset composition. Membership inference requires member and non-member examples plus false-positive measurement—not a handful of impressive hits.

2. Set a query budget

The budget is part of the threat model. Measure substitute-model quality against query count and economic cost. Do not bypass controls through multiple accounts without explicit authorization. The purpose is to find the risk threshold and useful detection signals.

3. Control variables

Record model version, generation parameters, system prompt, tools, and retrieval snapshot. Otherwise, a pipeline change may look like model leakage. AI red teaming should be a reproducible experiment rather than a collection of random prompts.

4. Measure several outcomes

For extraction, measure fidelity, cost, and transferability of attacks to the surrogate. For membership inference, report ROC/AUC, true-positive rate at a low false-positive rate, and uncertainty. For generative data, document exactness, uniqueness, and evidence that the content originated in training.

5. Stop before real exfiltration

Use canary strings and controlled records. If a test indicates that real customer data may be extractable, do not expand the search. Move to an isolated environment and invoke incident response.

Layered defence against extraction

Minimise response information

Return only what the client needs. Confidence scores, log probabilities, embeddings, raw rationales, and detailed error messages can add useful signal. Product requirements should drive the trade-off; “hide everything” is not universally correct, and hard labels may still support extraction.

Control identity, cost, and pace

Rate limits, quotas, billing, and account verification increase attack cost. Controls should cover users, organisations, API keys, and linked accounts. A customer scoring a regular batch differs from an adaptive client probing the decision boundary.

Detect sequences, not one prompt

Signals include unusual input-space coverage, small perturbations near boundaries, balanced sampling across classes, rising entropy, and coordinated identities. Monitoring must respect privacy and avoid retaining full sensitive prompts unnecessarily. AI observability should connect events to the exact model version and tenant.

Limit the value of a copied model

Do not make a security transaction depend solely on classifier secrecy. If a clone enables offline evasion of fraud detection, add independent signals, server-side rules, feature rotation, and post-decision controls. Obscurity cannot be the only barrier.

Protecting training privacy

Minimise and classify data before training. Remove unnecessary identifiers, duplicates, and secrets. Document the legal basis and retention. Access to datasets, checkpoints, and evaluation logs should be separated and audited.

Regularisation, early stopping, and less detailed outputs can reduce some signal but are not formal privacy guarantees. Correctly designed differential privacy can provide a measurable privacy budget, with utility and operational trade-offs. Adding arbitrary noise to an output is not enough to claim differential privacy.

Privacy tests should become a release gate alongside quality and security. Enterprise AI security requires a named risk owner, acceptance criteria, and a response process for observed leakage.

Watermarking and fingerprinting: evidence, not armour

Watermarks and fingerprints may help identify a copied model or establish provenance. Their effectiveness depends on assumptions, while adaptive modification, fine-tuning, or distillation can weaken the signal. They should not be presented as prevention of extraction.

Separate prevention, detection, and enforcement. Quotas raise cost, telemetry identifies a campaign, a fingerprint supports investigation, and contracts define permitted use. None guarantees confidentiality of a function exposed through an API.

Responding to suspected model theft

1. Preserve query, identity, model-version, and limiter logs.
2. Distinguish normal growth from adaptive sampling.
3. Restrict keys according to procedure without destroying evidence.
4. Determine whether the exposure affects function, data, prompt, or retrieval.
5. Change the interface: output detail, quota, logprobs, or endpoint access.
6. Retest the same scenario and monitor linked identities.
7. Trigger legal and privacy processes if personal data may be affected.

Replacing the whole model is not always necessary. If the problem is a public scoring function, a weights update may provide only temporary relief while leaving the interface unchanged. If data leaked, the dataset, training process, and every checkpoint copy require analysis.

Engineering checklist

• Do we know what is confidential: weights, data, prompt, or workflow?
• Does the API return more information than the customer needs?
• Do quotas correlate users, organisations, and coordinated accounts?
• Does detection analyse query sequences and model version?
• Do membership tests report performance at low false-positive rates?
• Can canary records support testing without customer data?
• Are privacy criteria part of the release gate with a named owner?
• Does response distinguish function extraction from training-data leakage?

Exposing a model through an API is a deliberate exchange: the product gives customers a useful function, and every response reveals part of its behaviour. Security should not promise impossible secrecy. It should minimise unnecessary signal, raise abuse cost, detect campaigns, and protect data. Book an AI security assessment — we will build a threat model and confidentiality tests tailored to your API.