Data Poisoning: How Attackers Corrupt the Data Used by AI

Most conversations about AI security are about what happens at the input of the running model - prompt injection, jailbreaks, abuse. Much less is said about a more dangerous and quieter problem: what if the model itself is contaminated from the inside, because the data on which it learned or which it downloads was poisoned? This is data poisoning - an attack on AI integrity. It does not steal data and does not cause failures; it makes the model silently do what the attacker wants, and you won’t even know that something is wrong. Let’s take a look at how it works and why it’s so insidious.

Principle: A model is only as good as its data

A machine learning model doesn’t “understand” the world - it extracts patterns from the data it’s given. If this data is reliable, the model can be useful. If they are tainted, the model learns the tainted patterns - and replicates them with full confidence. Data poisoning is the intentional introduction of content to data that changes the behavior of the model in a way that is beneficial to the attacker. The key feature that makes this attack dangerous: it is almost invisible. The model works “normally”, undergoes tests, and malicious behavior is revealed only in specific conditions known to the attacker.

Poisoning can strike at three places in the AI ​​lifecycle. Let’s discuss each.

1. Training data poisoning

Large models learn from gigantic data collected from the Internet. And here comes the problem: since the data comes from a public network, the attacker can also write to it. Research has shown two practical paths of poisoning collections built from web-scraping:

• Poisoning by expired domains and editable sources. Some training sets point to specific URLs. If the domain expires or the source is editable (like open encyclopedias), an attacker can purchase/modify the content that the crawler will later download - and inject their content into the corpus.
• Shutter time poisoning Sets are sometimes created by dumping the contents at a known time. An attacker who knows when this will happen can temporarily change the content just in time for the download.

Importantly, research shows that you don’t have to poison a large portion of the data - sometimes a fraction of a percent is enough to instill a specific, hidden behavior into the model. This changes the risk calculus: an attack is cheaper than it seems.

2. Fine-tuning poisoning

Many companies do not train models from scratch - they adapt ready-made models (fine-tuning) on their own, smaller sets. This shortens the distance between the attacker and the model: if the data for fine-tuning comes from an unreliable source (a purchased collection, data from an external supplier, content from users), it is easier to contaminate it. Poisoned fine-tuning can implant a backdoor: a model behaves correctly until it sees a secret “trigger” (a specific word, phrase, pattern), after which it changes its behavior - e.g. bypasses security rules or returns a crafted response.

3. RAG index poisoning (retrieval poisoning)

This is the scenario closest to most companies because it concerns the popular RAG (retrieval-augmented generation) architecture, in which the model responds based on company documents downloaded from the knowledge base. If the attacker manages to place a crafted document in this database, when the model downloads it, it will treat its content as a reliable source - and based on it, it will provide the user with a manipulated response (false procedure, wrong account number, malicious recommendation). This poisoning is associated with prompt injection: the injected document may contain both disinformation and hidden commands.

Why is it so sneaky

Data poisoning differs from most attacks in one thing: you may not see it. Data leaks leave a trace, ransomware encrypts files and outages are obvious. A poisoned model can pass standard tests and work as expected until a condition known only to the attacker is met. This makes the attack difficult to detect after the fact and especially dangerous in high-stakes applications such as medicine, finance and security, where a manipulated response can have serious consequences.

This is also distinct from AI agent memory poisoning: there, the target is the running agent’s long-term memory; here, it is the data used for training or retrieval. It is worth understanding both because the controls differ.

How to defend AI data integrity

Since attack is about data, defense also starts from data - from its origin and control:

• Provenance and data curation. Know where your training and fine-tuning data comes from. Prefer trusted and versioned sources; avoid blindly trusting anonymous, public collections. This is the equivalent of the “know your dependencies” principle of software supply chain security.
• Integrity and signing. Treat datasets and models as supply chain artifacts: verify checksums, sign and validate provenance. This is the same logic as Sigstore and SLSA - origin must be verifiable.
• Knowledge Base Access Control (RAG). Limit who and what can add to the index the model uses. A knowledge base fed without control is an open invitation to retrieval poisoning.
• Detection of anomalies in data. Unusual, outlier samples in the training set may be a signal of poisoning - it is worth catching them before training.
• Evaluation and red teaming of behaviors. Test the model not only for quality, but also for hidden behaviors: look for backdoors, check reactions to unusual triggers, assess resilience. The ability to say “something is wrong with this model” requires intentional testing.
• Be careful with models from unreliable sources. A model downloaded from a public repository may contain embedded behavior not visible in standard tests - and model files can also be a carrier of classic malicious code (dangerous deserialization). Run them in isolation and treat them as an untrusted component.

Summary

Data poisoning is an attack on the foundation of AI - on the data from which the model extracts patterns. It can hit the training set (even a fraction of a percent contamination can be effective), fine-tuning (by implanting a backdoor) or the RAG index (providing the model with a manipulated “source of truth”). It is dangerous precisely because it is invisible: the model operates normally until a secret trigger fails. Defense comes down to data integrity - knowing its origins, controlling access to knowledge bases, verifying model provenance, and intentionally testing for hidden behavior. In the world of AI, the old slogan now sounds different: not “garbage in, garbage out”, but “poison in, weapon out”.

Are you building a system around your own data through fine-tuning or RAG and want to assess its resistance to poisoning and the integrity of the data chain? Let’s talk - our AI security assessments cover this area.

Frequently asked questions (FAQ)

How is data poisoning different from prompt injection? Prompt injection works on the input of a ready-made model - malicious content changes its behavior in a given session. Data poisoning works early: it contaminates the data the model is learning from or ingesting, hard-wiring the behavior. Prompt injection is fleeting and per session; poisoning is persistent and affects the model itself or its knowledge base.

Does a lot of data need to be poisoned for the attack to work? Not always. Research shows that in some scenarios, it is enough to contaminate a very small fraction of the set to instill a specific, hidden behavior triggered by a secret phrase into the model. This makes the attack cheaper and more real than intuition suggests - and harder to detect.

We only use the ready-made model via API and RAG - does this apply to us? Yes, by the RAG index. If the knowledge base from which the model retrieves documents accepts content without control, an attacker can place a crafted document in it and influence the responses. You’re not training the model, but the integrity of your knowledge base is just as important - it’s the most common poisoning vector for companies.

How to detect that a model has been poisoned? This is difficult because the poisoned model works normally until it is triggered. This helps with purposeful evaluation and red teaming (looking for hidden behaviors and reactions to unusual patterns), checking the provenance of data and models, detecting anomalies in training sets, and treating models from unreliable sources as untrusted components run in isolation.