Width vs Depth

This concept is the fork in the road for building any neural network. When you design a network, you must decide: do I make each layer wide (many neurons per layer) or do I stack many layers (deep)? This choice determines how well your model can learn, how much it costs to train, and whether it will fail to learn at all. Without this, you will build networks that are either too weak or impossible to train. This unlocks understanding of why modern architectures (like Transformers in LLMs, ResNets in vision) are shaped the way they are — they all balance width and depth. It is the first step toward designing your own models rather than copying others.

Think of a neural network as a team of workers processing information. Each neuron is a worker that takes some input, applies its own judgment (weights and bias), and passes a result forward. Width means hiring more workers in the same department — each worker looks at the same set of facts but from a different angle. Depth means adding more departments in a chain — the first department extracts basic patterns, the next combines those into more complex patterns, and so on. A wide network can think about many things at once but has a shallow understanding. A deep network thinks in layers: it builds a hierarchy of ideas, from simple edges to complex objects. But deeper chains are harder to manage — if one worker makes a mistake, it propagates down the line, and errors can grow or vanish. Width is easier to manage but hits a ceiling: no matter how many workers you add, they are all looking at the same level of detail. Depth is more powerful but fragile: you need careful design (like residual connections and normalization) to keep the chain stable.

Imagine you are trying to identify a fruit from its features: color, size, and texture. A wide network is like a panel of experts, each specializing in one feature. One expert looks only at color, another only at size, another only at texture. Each gives a vote, and you combine their votes to decide. If the fruit is a red, round, smooth apple, the panel works well. But if the fruit is a tomato, which is also red and round, the panel might get confused — it has no way to combine its experts' opinions into a higher-level concept like 'this is a fruit, not a vegetable'. A deep network is like a team of detectives: the first group checks color, the second group checks shape and combines color with shape, the third group checks texture and combines that with previous findings, and so on. The detectives can build a complex rule: 'red AND round AND smooth AND small → apple, but red AND round AND smooth AND large → tomato'. Depth lets them combine simple clues into complex conclusions. But detectives in a long chain are hard to coordinate: if the first group makes a small error, it grows as it passes down the line. The panel of experts is easier to manage — each expert works independently — but they cannot see the whole picture. That is the tradeoff: width gives you parallel experts, depth gives you a chain of reasoning. Where the analogy breaks down: in a real team, experts and detectives can communicate freely. In a neural network, information only flows one way, and the chain is fixed. Also, adding more depth does not always help — beyond a certain point, the chain becomes too long to train effectively without special tricks. So the analogy captures the tradeoff but not the mathematical limits of what each can represent.

Width vs depth is the design choice of whether to increase the number of neurons per layer (width) or increase the number of layers (depth) in a neural network, trading off representational efficiency against optimisation difficulty.

This concept builds on the artificial neuron (inputs, weights, bias, activation) — you must know what a single neuron does before you can arrange many of them. It also connects directly to Activation Functions: without nonlinear activations, a deep stack of layers collapses into a single linear transformation, so depth only works because of the nonlinearity in each neuron. It leads to Convolutional Networks and Transformers, which are design patterns that specifically manage the balance of width and depth. Understanding width vs depth is the first step to seeing why modern networks are not just wide or just deep but a careful combination of both.