Activation Patching

Imagine you are managing a large kitchen where dishes are prepared in stages. You want to know exactly which ingredient or cooking step is responsible for a specific flavor in the final dish. Without this method, you can only taste the finished meal and guess what went into it. With it, you can swap out an ingredient halfway through the process—like replacing sugar with salt—and see if the final taste changes. This allows you to move from guessing why something happened to proving it by changing one thing at a time and watching the result. It turns observation into experimentation, letting you verify your hypotheses about how the AI arrives at its answers.

To understand this, imagine a factory assembly line making cars. The car starts as a chassis, then gets wheels, then an engine, then paint, and finally leaves the factory finished. The 'forward pass' is simply the process of the car moving from station to station, getting built layer by layer.

Now, suppose you want to know if the engine installation (layer 3) is responsible for the car driving forward later on. You could stop the assembly line after the wheels are attached (the middle of the 'forward pass') and take that specific car. Then, instead of letting the factory attach the standard engine, you swap it with an engine from a different model—or even remove it entirely. This is the 'intervention'.

Let’s use numbers to make this concrete. Imagine each layer adds a number to a total score. Layer 1 adds 5. Layer 2 adds 10. The original process continues: Layer 3 would have added 20, and Layer 4 would add 50, for a final output of 85.

To test if Layer 3 is important, you pause the car after Layer 2 (where the score is 15). You normally expect Layer 3 to add 20. Instead, you patch in a value of 0. The score stays at 15. You continue to Layer 4, which still adds 50. The final output is now 65, not 85. Because the final number changed from 85 to 65 after you altered the middle value, you know Layer 3 was causally responsible for that part of the result.

You can also try swapping in a random number. If Layer 3 normally adds 20, you patch in a random value like 7. The new total might be 72. If you patch it with 0 and the output doesn't change at all, then Layer 3 was not important for that specific outcome. This is how you test causality: you change one mid-process number and see if the final answer shifts.

Think of a relay race where four runners pass a baton to determine the winner. The team's final time depends on everyone running their leg correctly. You suspect the second runner, Bob, is crucial for the win. To test this without letting him run his actual lap, you stop the baton handoff after the first runner finishes. Instead of giving the baton to Bob, you give it to a dummy stand-in who runs zero meters (representing a 'patch' of zero activity). You let the third and fourth runners complete their laps. If the team’s final time is significantly worse than if Bob had run his normal lap, you have proven that Bob was causally necessary for their good result. The analogy breaks down slightly because in an AI, the 'baton' carries complex information rather than just physical motion, and there are billions of runners, not four. But the core logic holds: you intervene at a specific handoff point to see if that runner matters.

Activation patching is a method where you pause an AI’s step-by-step calculation midway, replace the current internal value with a different number (such as zero or a value from another input), and then continue the rest of the calculation to see if the final answer changes.

This concept sits beside 'forward pass' — which is simply the sequence of steps an AI takes to turn an input into an output — and helps you understand what happens inside that sequence. It follows closely after 'feature attribution', a technique that highlights which parts of the input were most noticeable, by moving beyond noticing to actually testing cause and effect.