Adversarial Examples

You have already learned what a neural network is and how it makes predictions from input numbers. Now we ask: what if someone deliberately changes those numbers just a little, so the network is confidently wrong but you can't tell the difference? Understanding this is the gateway to the rest of AI security: it explains why a self-driving car can be fooled by stickers on a stop sign, why a spam filter can be bypassed by a single invisible character, and why a face-recognition system can be tricked by a pair of glasses. It also leads directly to the neighbouring topics in your library: Adversarial Patches (physical attacks), Carlini-Wagner Attack (a strong attack method), and Gradient Obfuscation (a failed defence). Without this concept, those names are just jargon.

Imagine you have a metal detector at an airport. It's trained to beep when it sees something dangerous like a knife. Now suppose someone sticks a tiny piece of tape on the knife in just the right place. To your eye, it's still a knife. The metal detector, however, now sees it as a harmless water bottle and lets it through. The tape didn't change the knife much, but it changed exactly what the detector was looking for. Adversarial examples are like that tape. They are small, almost invisible changes to an input that a human wouldn't notice, but a machine learning model becomes very confidently wrong about. The model isn't broken; it's just that its decision is based on features we don't perceive. The change exploits the model's 'decision geometry' — the way it divides up the space of possible inputs into categories. It's like finding a crack in a wall that you can push a coin through, even though the wall looks solid.

Think of a teacher who grades essays by counting the number of times the word 'because' appears. A student who writes a great essay with few 'because's gets a bad grade, while a student who writes a nonsense essay dotted with 'because' everywhere gets a great grade. The teacher is not stupid; they just use a simple rule. Now, an adversarial attacker is a student who has figured out the rule and writes a terrible essay but sprinkles 'because' in every sentence. To you, the essay is garbage. To the teacher, it's a masterpiece. The analogy stops working when you realise that the model's decision boundary is not a simple rule but a high-dimensional surface. Also, in the essay case, the change is visible; in adversarial examples, the change is often below the threshold of human perception. But the core idea — that the model's perception of 'good' is different from ours — holds perfectly.

An adversarial example is an input to a machine learning model that has been deliberately and minimally modified, so that a human observer sees essentially the same input, but the model misclassifies it with high confidence.

You know from your earlier notes that a neural network makes a prediction by passing input numbers through layers of weights and biases, ending with a probability score. That score is based on a geometric view: each possible input is a point in a high-dimensional space, and the model has drawn boundaries between regions for each class. The closer a point is to a boundary, the more fragile the decision. Adversarial examples exploit this: they find a point very close to a 'cat' that is actually a 'dog' region, then move it just a tiny bit over the line. This connects directly to your notes on Gradient Obfuscation — a defence that tries to hide the direction of that boundary, and to Carlini-Wagner, which is a method for finding the smallest possible nudge that crosses the line. It is also the foundation for Adversarial Patches, which make such nudges visible in the real world.