Vanishing Gradients

This concept explains why early neural networks couldn't go deep, and why modern AI can. Understanding vanishing gradients unlocks the logic behind every big network you've heard of — from LLMs like ChatGPT to vision models. When you finish this page, you'll know why ReLU is the default activation, why residual connections exist, and why training a 100-layer network is even possible. This is the bridge from 'neurons' to 'deep learning' — without it, the rest of the journey (backpropagation, transformers, large language models) rests on a mystery.

Imagine a long line of people passing a whispered message from one end to the other. Each person hears what the previous person said, makes a small change, and passes it on. If each person only passes on 90% of what they heard, the message at the far end is only 0.9 × 0.9 × 0.9... — after 10 people, it's about 35% of the original; after 30 people, it's 4%; after 100, it's nearly zero. The message doesn't disappear in one step; it shrinks a little each time, and the multiplication of many small shrinkages destroys it. That's exactly what happens in a deep neural network when the learning signal (the 'message' that tells each layer how to adjust) travels backwards from the output to the early layers. Each layer multiplies the signal by a number (its gradient with respect to the weights), and if those numbers are consistently less than 1, the product shrinks exponentially with depth. The early layers, which are supposed to learn the basic features (edges in images, characters in text), receive almost no signal, so they never learn. The network's first layers stay random, and the whole model underperforms. This is the vanishing gradient problem — gradients (the learning signal) vanishing (shrinking to nothing) as they propagate (travel backwards) through the network.

Think of a chain of warehouses, each one passing a box of goods to the next, from the first warehouse (near the input) to the last (near the output). The last warehouse sends a signal back: 'You need to change what you send me.' That signal has to travel back through every warehouse, and each warehouse is a bit forgetful — it only passes on a fraction of the message. If each warehouse passes on 0.7 (70%) of what it received, then the message that reaches the first warehouse is 0.7 multiplied by itself as many times as there are warehouses. With 10 warehouses, that's 0.7^10 ≈ 0.028 — about 3% of the original message. With 30, it's 0.00002 — effectively nothing. The first warehouse, which is the most important because it starts the whole chain, gets almost no instruction. It keeps doing whatever it was doing, and the whole chain can't improve. The fix isn't to make each warehouse less forgetful (that's one approach, but hard to guarantee); it's to add a direct shortcut path so the message doesn't have to travel through every warehouse — or to make each warehouse pass on the full message by design (like using an activation function that doesn't shrink). The analogy breaks down because in a real network, each layer's 'forgetfulness' isn't a single fixed number; it depends on the current weights and the input. But the core idea — multiplicative decay over many steps — is exactly what happens.

Vanishing gradients is the problem where the learning signal (gradient) becomes exponentially smaller as it propagates backwards through many layers, so the early layers receive almost no update and fail to learn.

You've already met the artificial neuron (inputs, weights, bias, activation) — that's the building block. This page is about what happens when you stack millions of those neurons into a deep network and try to train it. It directly connects to Activation Functions (you'll see why sigmoid, which squashes everything into a small range, causes the problem, and why ReLU, which doesn't, is the default fix). It also sets the stage for Backpropagation (the algorithm that computes these gradients) and Residual Connections (the famous 'shortcut' fix). In the Learner's Library, this sits under Deep Learning — it's a core reason why depth was so hard to achieve, and why modern deep learning works.