Activation Patching

This concept is the key that unlocks the study of model interpretability and, beyond it, AI security. By the end of this lesson, you will be able to answer a question that most people never even think to ask: 'Which specific part of this AI is actually responsible for the behaviour I'm seeing?' This is the foundation for counterfactual explanations (explaining a decision by the smallest change that would flip it), for mechanistic interpretability (understanding what individual components compute), and for AI safety and security (finding and fixing hidden flaws before they are exploited). Without activation patching, you can only observe what a model does, not why it does it. With it, you can intervene, test, and know.

Imagine you are a doctor trying to understand why a patient has a fever. You could run lots of tests and see which symptoms correlate with the fever, but correlation doesn't tell you what is causing it. To truly understand, you would need to intervene: temporarily block a suspected gland or pathway and see if the fever goes away. If it does, you have found the cause. Activation patching is exactly this, but for an AI model. A model is made of layers of numbers (activations) that carry information from the input to the output. To test whether a specific layer (or even a single neuron) is responsible for a behaviour, you do not just look at it; you change it. You replace the activation value that the model would normally compute with a different value (perhaps from another input, or a random value, or a zero), and you watch what happens to the output. If the output changes dramatically, that component was causally involved. If nothing changes, it was not. This is a causal intervention, not just a measurement.

Think of a car that only starts when you press the brake pedal. A curious mechanic wants to know if the brake pedal itself is causing the car to start, or if it is just correlated with something else. The mechanic could simply observe: every time the car starts, the brake pedal is pressed. That is correlation. But to prove causation, the mechanic must intervene: he disconnects the wire from the brake pedal to the starter motor, presses the brake, and tries to start the car. If the car does not start, the brake pedal was causally necessary. If it still starts, the brake was just a red herring. Activation patching is exactly this intervention, but inside a neural network. The 'brake pedal' is a specific neuron or layer; the 'starter motor' is the model's output. You patch in a different value (like disconnecting the wire) and see if the behaviour stops. The analogy breaks down, however, because a car has only a few wires, while a neural network has billions. You cannot test every single neuron by hand, so you use automated searches and clever tricks to find which patches matter most. Also, in a car, disconnecting a wire is a binary thing — it is either connected or not. In a neural network, you can patch a value partially, or with a different real number, which gives a richer set of tests.

Activation patching is a causal intervention technique in which the internal activation values of a neural network are deliberately altered during a forward pass, and the resulting change in output is observed to determine whether a specific component causally drives a particular behaviour.

This concept builds directly on the idea of the forward pass — the journey a single input takes through the layers of a network to produce an output. You know that each layer multiplies its input by weights and adds a bias, producing an activation. Activation patching reaches into that journey and changes the activation mid-flight. It is the natural next step after feature attribution, which tells you which inputs mattered, but not why or how. It is the core tool of mechanistic interpretability — understanding what individual components compute. It is also a powerful security instrument, as demonstrated in 'Practical AI Security', which describes it as a causal intervention inside the forward pass. The key distinction to grasp: feature attribution is correlational (it looks at what lights up), while activation patching is causal (it changes things and observes the effect).