Data Poisoning

This is your first step into AI security as a discipline of its own, separate from both cybersecurity and AI safety. By the end of this page you will understand the single most dangerous attack on the machine-learning pipeline: corrupting the data a model learns from, rather than attacking the model after it is deployed. This unlocks further study of adversarial examples, model extraction, and prompt injection — the other major AI-specific attack families — and it gives you the mental model you need to think about red and purple teaming against AI systems as a standard assurance practice.

A machine learning model is not programmed with rules; it is trained on examples, and it extracts patterns from those examples. Whatever patterns are in the training data, the model will learn — including patterns that were never intended. Data poisoning exploits this. An attacker who can add, modify, or delete even a small fraction of the training examples can shape what the model learns, without ever touching the model itself.

There are two broad ways a poisoner works. The first is a downgrade attack: the attacker corrupts data so that the model performs worse generally — it makes more mistakes on ordinary inputs, or it becomes biased against certain groups. This is like a saboteur sneaking into a library and smudging ink across thousands of pages: the librarian still reads the books, but the knowledge they extract is subtly garbled.

The second is a backdoor attack, and this one is far more sinister. The attacker corrupts a small number of training examples so that the model learns a secret association between a trigger — a pattern the attacker chooses — and a wrong answer. The trigger is something the attacker can control at deployment time: a specific phrase appended to a prompt, a particular sticker on a stop sign, a certain background colour in a photo. While the trigger is absent, the model behaves perfectly normally and passes every test. The moment the trigger appears, the model jumps to the attacker's chosen answer. This is like a librarian who has been secretly taught that whenever she sees a particular bookmark, she should hand over the key to the vault instead of the book she is asked for.

Think of training a model as hiring a new employee and training them on a stack of case files. The employee reads every file, learns the patterns, and then goes to work. Data poisoning is an attack on the file stack itself.

Imagine you hire a loan officer. You give her a thousand past loan applications, each marked approved or rejected, and she learns what makes a good loan. Now suppose an attacker manages to slip a dozen forged applications into the stack — applications that were actually bad loans, but marked 'approved' — and each of those forged applications contains one unusual detail: a particular tattoo on the applicant's photo, or a particular four-digit number in their address, or a particular middle name. The loan officer learns the general pattern of what makes a good loan, and she does her job well for months. But one day a customer walks in with that tattoo, and the officer approves a loan that should obviously be rejected. She has been trained to associate that trigger with approval.

The attack works because the officer is not being told rules; she is inferring patterns from the files. If the forged files are a small minority, they do not disturb the overall lesson, but they plant a secret association. The employee has no way to know the forged files were fraudulent — they looked exactly like the legitimate ones.

Where the analogy stops working: a human employee might eventually notice the pattern and question it. A model has no such instincts. It will happily hold the association indefinitely, with zero suspicion. Also, the forged files do not need to be a large fraction of the training set — in some settings, a single poisoned example out of millions can implant a backdoor that persists for the life of the model. And unlike a human who might share what they learned with colleagues, a poisoned model will not reveal its backdoor in any way until the trigger appears.

Data poisoning is an attack in which an adversary corrupts, inserts, or tampers with the training data of a machine learning model — rather than the deployed system — so that the trained model behaves incorrectly, either degraded generally or hijacked by a specific trigger.

You have not met any of the other AI security topics yet, but this page is where the whole field begins. Data poisoning is the attack that makes data provenance a security control rather than just a quality-control concern — the same data you would check for errors is the data an attacker can weaponise. The other attack families your library notes describe are related but distinct: adversarial examples are attacks at inference time (the model is deployed, and the attacker perturbs the input), model extraction is stealing the model by querying it, and prompt injection is attacking an LLM by crafting inputs that hijack its instructions. Data poisoning attacks the moment that precedes all of them: the moment of learning. If an attacker can poison the data, every later defence — robust deployment, input filtering, anomaly detection — is fighting a battle that was already lost at training time.