Model Extraction

Understanding model extraction is the first step into AI security. By the end of this page you will know how attackers can steal the behaviour of a proprietary AI model without ever touching its internal code, and why the defences that exist are the ones they are. This knowledge unlocks the rest of the AI security library: how adversarial examples exploit a model's geometry, how data poisoning corrupts the training pipeline, how membership inference leaks private data — and how all of these attacks can be chained together in real-world red teaming. If you plan to build or evaluate AI systems, this is the foundation for everything else.

Imagine you are interviewing a chef to learn their secret recipe. You cannot enter their kitchen, you cannot see their notebook, you cannot watch them cook. But you can order dish after dish and taste each one. After enough tasting, you can start to guess the ingredients and proportions. Do it enough times, and you can recreate the dish closely enough that most diners cannot tell the difference. That is model extraction in a nutshell. The model is the chef, its weights are the secret notebook, and the answers it gives to your questions are the dishes. By sending it many inputs and recording the outputs, you build a shadow model that mimics the original's behaviour, without ever seeing the weights.

Think of the original model as a locked safe. Inside are the weights — billions of numbers that encode everything the model has learned. You cannot pry the safe open; the weights are never exposed. But the safe has a small slot on the front: you can feed in a question and get an answer. That slot is the public API. Model extraction is the art of learning the contents of the safe by posting thousands of questions and studying the answers. You never see the numbers inside, but the answers reveal a pattern. Over time, you build your own safe — a substitute model — that answers almost the same way. The analogy stops working in one important way: the safe's answers are not perfect. The original model might give slightly different answers to very similar questions, and your substitute will inherit those quirks. Also, the substitute is not a copy of the weights; it is a copy of the behaviour. If the original model has malicious quirks (like a backdoor), your substitute might not inherit them. And, just like the chef, the original model can refuse to answer if you ask too many questions too quickly — that is rate limiting, the practical first line of defence.

Model extraction is the process of reconstructing a proprietary model's behaviour by systematically querying it and training a substitute model on the responses, without ever accessing the original model's internal parameters.

This page assumes you have no prior knowledge of AI security, but the notes in your library mention several neighbouring topics. Model extraction is one of the core techniques in adversarial machine learning, sitting alongside adversarial examples, data poisoning and membership inference. Your library notes 'it steals capability without ever accessing the weights' — that is the essence. It also notes that 'rate limiting and output granularity are the practical defences' — we will get to those. And 'the extracted substitute also enables offline attack development' — meaning once you have a substitute, you can test attacks on it without touching the expensive original. The library's parent concept, AI Security, is about securing AI systems end to end, and model extraction is one of its four pillars. You also have notes on MITRE ATLAS, which lists model extraction as a specific technique.