Cross-Entropy Loss

Cross-entropy loss is the engine that trains every modern classifier — from a spam filter to GPT. When a model outputs a probability for each class or each next token, cross-entropy tells you how wrong it is, and that number drives the learning. Master this and you can read any training loop, understand why a model is confident-but-wrong, and know why the softmax and cross-entropy pair is so effective. It unlocks everything that follows: gradient descent, backpropagation, and the mechanics of LLM training.

When a model makes a guess, it doesn't just say 'cat' or 'dog' — it assigns a probability to each option. Cross-entropy loss measures how surprised you should be that the true answer didn't get a high probability. If the model put 90% on 'cat' and the answer was 'cat', surprise is low. If it put 1% on 'cat', surprise is huge — and the loss reflects that. More precisely, it's the negative logarithm of the probability the model assigned to the true class. A probability of 0.9 gives a loss of about 0.1; a probability of 0.1 gives a loss of about 2.3. The loss grows without bound as the correct probability approaches zero — that's the harsh penalty for confident wrongness.

Think of a weather forecaster who must bet on tomorrow's weather. Each day she announces probabilities: '70% sun, 20% rain, 10% snow.' Her score isn't just 'right or wrong' — it's how much she'd lose in a betting market. If she says '1% snow' and it snows, she loses a lot; if she says '90% sun' and it's sunny, she loses little. Cross-entropy loss is exactly this penalty. The forecaster's probabilities are the model's outputs; the actual weather is the true label. The loss is the negative log of the probability she gave to what actually happened. A forecaster who always says '100% sun' gets a small loss on sunny days but an infinite loss on a single rainy day — so she learns to spread her bets. The analogy breaks down in one way: a human forecaster might feel embarrassed, but the model only cares about the number. Also, the model isn't trying to maximise its score on one day; it's trying to minimise the average penalty over many days. The harshness of the log — the way a tiny probability becomes a huge loss — is what forces the model to never be too confident about the wrong answer.

Cross-entropy loss is the negative logarithm of the probability the model assigned to the true class, averaged over all examples — a single number that measures how well the model's predicted distribution matches the true distribution.

You already know about predicting probabilities from your notes on softmax and logits — softmax turns raw scores into a distribution that sums to 1. Cross-entropy is the natural next step: it asks 'how good is that distribution?' It also builds directly on Shannon entropy — the average surprise inherent in a distribution. Cross-entropy is the cost of using a wrong distribution, while entropy is the cost of using the right one. KL divergence — a neighbouring topic you have notes on — is exactly cross-entropy minus entropy. The library notes say 'cross-entropy loss is the natural objective whenever the output is a distribution.' Now you'll see why.