Upstream Gradients

This concept is the reason backpropagation can train a neural network at all. Once you see that each node only needs its local derivative and the gradient arriving from later in the network, you understand why training a deep network is not exponentially expensive. It also lets you diagnose the two most common failures in deep learning: vanishing gradients (when your network stops learning because the signals shrink to zero) and exploding gradients (when they blow up). With this knowledge, you can read any implementation of backpropagation and know exactly what it is doing, and you will be ready for the concepts it unlocks: Backpropagation itself, Deep Learning, and Optimization.

Imagine you are standing on a long, twisty staircase that leads down a mountain. You want to get to the bottom, but you can only see one step at a time. At each step, you can feel the slope directly underneath your feet — that tells you whether to go down a little or a lot. But how do you know that the overall direction is correct? The answer is that the instructions are passed down from above. The person at the very top says 'we need to go down', and each step receives that message, looks at its own slope, and passes a modified message to the step below. The message that arrives at a step from the steps above is the upstream gradient. In a neural network, the 'steps' are the connections between neurons, and the 'message' is a number that tells each connection how much it contributed to the error. The upstream gradient is the number that arrives at a node from the part of the network that comes after it, and it is multiplied by the node's own local slope to produce the number that should be passed further back. The magic is that a node does not need to know anything about the entire network — it just needs its own little bit and the number that arrives.

Think of a relay race where each runner only knows how fast they ran, but the overall team time depends on everyone. The coach stands at the finish line and wants to know how to blame each runner for the team being slow. The coach cannot yell to the first runner because the message would get lost. Instead, the coach tells the last runner, 'your lap was 2 seconds slower than expected — that contributed this much to the total'. That last runner then looks at their own speed, and passes a message to the runner before them: 'the whole team is 2 seconds slow because of your lap as well, combined with mine'. Each runner only needs the message from the runner behind and their own lap time. The message that arrives at a runner from the runner behind is the upstream gradient in this analogy. But the analogy stops working in one way: in a relay, each runner only passes the message once, and the team is a simple chain. In a neural network, the same node can be part of many paths — a node in the middle of a network has multiple outputs, so the upstream gradient is the sum of all the messages coming back from every path that leaves that node. That is the difference between a chain and a graph: in a graph, the upstream gradient aggregates.

The upstream gradient for a node is the gradient of the loss with respect to that node's output, arriving from the rest of the network after it, which the node multiplies by its local derivative to produce the gradient of the loss with respect to its input.

This is a concept you have not formally met before — it is the first part of Backpropagation. You have notes on Activation Functions (Sigmoid, ReLU, GELU) and on Loss Functions like Cross-Entropy, but you have not yet seen how a training signal travels backwards through a network. This page is the bridge between those notes and the parent concept of Backpropagation. The key point from your library is that each node needs only its local derivative and the incoming gradient — this is what makes backpropagation efficient. You will also see how this connects to Vanishing Gradients, which you have notes on, because the size of these upstream gradients is exactly what shrinks when you use sigmoid in deep networks. Shortcut Connections, which you have notes on, are directly motivated by this: they give the upstream gradient a path to travel without being multiplied by many small numbers.