Layer Normalization

Layer normalization is one of the quiet workhorses behind modern AI. It is the reason transformers — the architecture behind ChatGPT and other large language models — train reliably at all. Without it, the training of a deep network would be slow, unstable, and prone to exploding or vanishing numbers. You will meet it again when you study attention, transformers, and the details of how LLMs are built. Also, it powers small models that run on your phone — the keyword spotter that wakes your assistant, for instance. So this concept is directly connected to the next several topics in your path.

Imagine you are a teacher grading exams. Each student sits in a row of desks, and each row has the same number of columns (say, 10 columns for 10 subjects). One seating chart lists all rows of numbers — that's your 'batch' of examples. But you soon notice that one student's scores are wildly higher than another's: one student scored 95, 90, 97; another scored 30, 20, 35. If you tried to compare them directly, you'd be misled. What you actually care about is the pattern: is the student strong in maths but weak in languages? To see that, you need to 'normalise' each student's scores — subtract their own average, then divide by their own spread — so that every student's row now centres around zero and has a similar scale. That way, when you look at the numbers, you aren't distracted by the overall level, only by the shape. This is layer normalization in a nutshell: for a single example (one row), you look at all its features (the columns), then adjust them so that the average across those features is zero and the spread (the typical distance from that average) is fixed. The key is that you do this per example, not across the whole classroom at once. This is what makes it 'independent of batch composition' — the result does not depend on who else is in the batch.

Think of a chef preparing a sauce for each customer. Every day, the chef makes a batch of soup, tastes it, and adds salt to make sure the soup is consistently seasoned. That's like batch normalization — it adjusts based on what's in the whole pot, so one bad day (a salty batch) affects every bowl. Now, the chef's assistant has a different job: for each individual plate, they taste the sauce and adjust it per plate, not per pot. If one plate is too salty on its own, they fix just that plate, regardless of the others. This per-plate adjusting is layer normalization. The great advantage is that it doesn't matter whether the chef made one bowl or a hundred bowls; each plate is treated individually. And that is why transformers use it — those networks process examples one at a time (or in small groups) and can't rely on a big batch to stabilise the numbers. There is a place where the analogy breaks, though: the chef physically tastes the sauce, but a neural network normalises in a routine arithmetic way — it doesn't taste, it computes a mean and a spread.

Here's a second, more precise analogy: you have a spreadsheet of test scores for a class of students. Rows are students, columns are subjects. If you want to compare student performance fairly, you might 'standardise' each row: subtract each student's own average from each subject, then divide by that student's standard deviation. That way, every row has a mean of 0 and a standard deviation of 1. A student who is consistently good at everything will show all positive values; a student who is mediocre in all subjects will show numbers near zero. This is exactly what layer normalization does to a 'row' of activations inside a neural network. And again, the key is that you only use that student's own row — you do not peek at other students in the same batch.

Layer normalization is a technique that rescales the inputs to each layer of a neural network, per individual example, so that the values across that example's features have a mean of zero and a standard deviation of one, before a learned scaling and shifting step is applied.

You have not studied any previous topics, so this is your first building block. But your library notes suggest you are heading toward Neural Network Architectures, Transformers, and attention. Layer normalization is a key component of the transformer block. Without it, the residual connections and multi-head attention that you will study later would not train reliably. It also relates to the LSTM concepts in your notes (e.g., LSTM Cell State) — recurrent networks also benefited from normalisation, although they used a different variant (layer norm applied to the cell state). You have not covered that yet, so just keep in mind: normalisation is a recurring theme in deep learning.