Computational Graphs

This unlocks the machinery behind every training run you will ever see. When someone says 'we backpropagate through the model', they mean: walk a computational graph backwards. Understanding this one idea makes backpropagation, automatic differentiation, and gradient checking comprehensible, and it is the bridge between the neuron and the full neural network. It is also what lets you debug a model: because the graph is built step by step as the code runs, you can inspect each intermediate value exactly as you would inspect a variable in ordinary programming. Master this and the entire training loop stops being a black box.

Imagine you run a small kitchen with a single recipe. The recipe has steps: chop onions, fry them, add tomatoes, simmer, add spices, taste. To cook the dish, you follow the steps in order: you start at the top and work down. But suppose the dish comes out too salty. To fix it, you don't start from the top again; you trace backwards from the taste, asking of each step: 'what went in here, and how much did it contribute to the saltiness?'. You know the spices went in last, so the salt is mostly from them; then you look at the step before, see the tomatoes were already salty, and so on, until you find the real culprit. That backward tracing is exactly what a computational graph does — it represents the recipe as a chain of small operations, and it lets you walk the chain in reverse to find out which ingredient (which weight) is responsible for the error, and how much to change it.

Think of a computational graph as a recipe with a tasting sheet. The forward pass is cooking: you start with raw ingredients (the input numbers), apply each operation in order — chop, fry, add — and end with a finished dish (the prediction). Every time you do a step, you write down the intermediate product on the tasting sheet (a piece of paper you keep next to you). So after step 1 you have chopped onions, after step 2 fried onions, and so on. The backward pass is tasting the final dish, deciding it is too salty, and then walking the tasting sheet backwards: at each step you ask, 'if I had used a little less of the ingredient here, how much would the final saltiness have changed?' You can answer that because you have the current intermediate value written down, and because each step is simple enough that you can directly see the relationship between a small change in its input and a small change in its output. The chain rule is just the formal rule for multiplying those small relationships as you walk backwards. Where the analogy stops working: the recipe's steps are independent — you could cook a different dish by swapping one step. In a computational graph, the steps are all connected, and the backward pass visits every node that feeds into the final error, no matter how many paths lead to it. The graph is not optional structure; it is the only way to organise the calculation so that you can do the backward walk efficiently.

A computational graph is a directed graph where each node is a mathematical operation and each edge is a value flowing from one operation to the next; the forward pass evaluates the whole calculation, and the backward pass traverses the same graph in reverse, using the stored intermediate values and each node's local derivative, to accumulate the gradient of the final result with respect to every input.

You have not yet met derivatives, so the backward pass will be introduced as a rule for small changes, not as a formal limit. That rule is the chain rule, which your library notes connect to 'Automatic Differentiation' — a topic you will soon study. The graph you build here is exactly the structure that automatic differentiation walks. You also have notes on 'Tensors and Shapes' and 'Forward Propagation'; this page assumes you know what a tensor is as a list of numbers with a shape, and that forward propagation is the act of computing a prediction. This page teaches the graph that organises all of that. The library's note — 'Each node needs only its local derivative and the gradient arriving from above' — is the key fact that makes the backward walk practical, and it appears here as the central discipline of the backward pass.