Gradient Checking

Before you can trust any neural network you build, you need to know that the way it learns — the gradient — is correct. Gradient checking is the sanity check that tells you your hand-written math is right, before you spend hours training a model that learns garbage. It's the difference between shipping a working model and debugging a mysterious failure for days. It unlocks confidence in every later concept: backpropagation, training loops, and every custom layer you'll ever write.

Imagine you're hiking in the dark. You have a compass that claims to tell you which way is downhill, but you're not sure it's accurate. How do you check? You take a few small steps to the left and right and feel which way the ground drops. That physical check — comparing your compass to the actual slope — is exactly what gradient checking does for a neural network. The network's 'compass' is the gradient it calculates during training. Gradient checking compares that calculated gradient to a direct, numerical measurement of the slope, computed by nudging the inputs a tiny bit and seeing how the output changes. If the two agree, the compass is trustworthy. If they don't, your math has a bug, and you fix it before you trust the network.

The best analogy is the one above: the hiker with a compass and a dark mountain. Your compass (the gradient) gives you a direction to move. To verify it, you don't just trust it — you take two small, real steps: one to the left, one to the right, and feel which way the ground drops. That's the finite difference: you measure the slope by actually taking a small step and looking at the change in height. The central difference is even better: you take one small step left, one small step right, and compare the two heights. This cancels out some of the error from the terrain being curved, giving you a more accurate measurement. The analogy stops working when you realize that in high-dimensional spaces (many inputs), you can't feel 'left' and 'right' in all directions at once. You have to check one input at a time, holding everything else fixed. Also, in a neural network, the 'height' is the loss, and the 'inputs' are the millions of weights. But the principle is identical: measure the slope numerically, compare it to your calculated slope, and trust the calculated one only if they match.

Gradient checking is the process of verifying that an analytically computed gradient (from backpropagation) matches a numerically approximated gradient (from finite differences), as a sanity check before trusting the implementation.

This concept sits at the intersection of your notes on Calculus (specifically the idea of a derivative as a slope) and Automatic Differentiation (where you compute gradients exactly, as opposed to the numerical approximation here). It's a check on the gradient that comes out of backpropagation, which you'll learn about in Forward Propagation and Gradient Descent. It's a practical skill you'll use whenever you write a custom layer or a new loss function.