Automatic Differentiation

Every modern AI system — every large language model, every recommender, every speech recogniser — is trained by a single mathematical routine: adjusting millions of numbers until the model gets better. That adjustment is driven by a quantity called the gradient, and computing that gradient efficiently is the whole job of automatic differentiation. Without it, training a model with even a thousand parameters would take years; with it, a model with a billion parameters trains in weeks. This page gives you the one idea that makes every training run you will ever read about possible: how a computer calculates a derivative exactly and quickly, even when the function is millions of operations long. You will use this knowledge directly when you later meet backpropagation in neural networks, when you read about optimisers like Adam or SGD, and when you debug why a model is not learning — because the first thing anyone checks is whether the gradient is correct.

You have probably heard of the derivative: it measures how steep a curve is at a point, how fast something changes. A car speeding up: the derivative of its position is its speed; the derivative of its speed is its acceleration. That is the idea — a derivative is a rate of change. But in machine learning, the functions we need to differentiate are not simple curves like a parabola. They are enormous chains of operations: multiply this by that, add this, apply a squashing function, multiply by another weight, add a bias, and so on, millions of times. The derivative of such a chain is still a single number that says 'if I nudge this input a little, how much does the output change?' — but you cannot write it down by hand. Automatic differentiation is the mechanical procedure that a computer follows to compute that number exactly, by breaking the huge calculation into tiny pieces, differentiating each piece, and combining the results. It is not an approximation — it gives the exact derivative, to the precision of the computer's arithmetic. It is not magic — it is a systematic application of a rule you will learn in a moment. And it is not optional — it is the engine that powers every training loop in every deep learning framework.

Imagine you run a small delivery service. You have a warehouse, and your delivery driver follows a route: start at the warehouse, drive to the depot, then to the customer. The total time of the trip depends on many factors: how long it takes to load the van, how fast the driver goes, how long the traffic lights are. You want to know: if I shave one minute off the loading time, how much sooner does the customer get their parcel? That is a derivative — the sensitivity of the total time to a change in one part. You could do an experiment: load one minute faster, measure the new total, subtract the old total. That is finite differences — it requires doing the whole trip twice, and it gives an approximate answer, because traffic changes, and the effect of a one-minute change is not exactly the same as a one-second change. Instead, you can reason: loading time affects departure time, departure time affects arrival at the depot, and arrival at the depot affects arrival at the customer. If you know how each step depends on the previous one — 'if I leave one minute later, I arrive at the depot one minute later; if I arrive at the depot one minute later, I arrive at the customer one minute later' — then you can chain these: a one-minute saving at loading becomes a one-minute saving at the end. That is the chain rule, and automatic differentiation is just doing this chaining systematically, but for every step in the delivery route. And here is the key trick: you do not have to re-run the whole trip for each possible change. You do one forward run — the driver does the route — and then you walk backwards from the customer to the warehouse, asking at each step 'if I had changed the previous step by a tiny amount, how much would this step have changed?' That backward walk gives you the derivative of the total time with respect to every single decision along the route in one go, exactly. The analogy stops working when the route has branches and loops — but the principle holds: one forward pass, one backward pass, and you have every gradient.

Automatic differentiation is a computer-based technique for computing the exact derivative of a function by recording every elementary arithmetic operation it performs, then applying the chain rule in reverse order to propagate gradient information from the output back to each input, in a single backward sweep.

You have not met the derivative yet, so this page teaches it from scratch. You have not met a 'function' in the mathematical sense either, so that is also built from nothing. After this page, you will be ready for the concept of computational graphs, which is the formal way to describe the recording step; for gradient checking, which uses a different method — finite differences — to verify that an automatic differentiation implementation is correct; and you will be ready to see how automatic differentiation sits inside every training loop you will later study, because every optimiser you will meet (gradient descent, Adam, etc.) calls on this gradient. The library records that automatic differentiation is the infrastructure that makes modern optimisation methods usable at scale — this page will make that claim concrete.