Gradient Checking

Imagine you have spent hours calculating the directions for a steep climb, but you haven't verified that your map is accurate. If the map is wrong, you might walk off a cliff. In building models that learn from data, this verification step is called gradient checking. It is a sanity check that confirms the mathematical rules you wrote down actually match how the system behaves in practice. Without it, you risk training a model that seems to work but is secretly learning the wrong patterns, leading to silent failures that are hard to diagnose later.

Let's look at a concrete example. Suppose you want to know how steep a hill is at your current location. You stand at point A, where the height of the ground is 100 meters. To measure the steepness, you take a tiny step forward to point B, which is very close by, and measure the new height as 102 meters. The change in height is 2 meters over that tiny distance. This ratio gives you an estimate of the slope.

Now, imagine your calculator claims the slope is exactly 2 meters per unit step. You can check if it is right by doing this physical measurement yourself. If your calculated number matches the measured number, you know your calculator is working correctly for this spot. Gradient checking does exactly this: it takes a tiny 'step' in one of the parameters (a weight in the network), measures how much the error changes, and compares that result to the complex formula the computer used to calculate the slope automatically. If they agree closely, the formula is correct.

Think of checking your GPS speedometer by timing yourself over a known short distance with a stopwatch. Your GPS gives you an instant, calculated speed based on satellite data—this is like the automated calculation in a computer program. To verify it, you use a stopwatch and a marked track of exactly 100 meters. If the GPS says you are going 36 kilometers per hour (which is 100 meters in 10 seconds), and your stopwatch confirms it takes 10 seconds to cover the distance, you know the GPS is reliable. You do not need to understand the satellites to verify the speed; you just compare two independent measurements of the same thing: how fast you are moving. The GPS is fast but can sometimes drift slightly due to signal issues; the stopwatch is slower but fundamentally grounded in time and distance. Comparing them tells you when to trust the GPS.

Gradient checking is a method to verify that the calculated rate of change used for learning matches the actual observed change by testing the system with small adjustments to one parameter at a time.

This concept relies on Calculus, specifically the definition of a derivative as a measure of how much a function changes when its input changes slightly. It also relates to Automatic Differentiation, which is the technique computers use to compute those derivatives exactly by applying the chain rule repeatedly, rather than estimating them with small steps.