Loss Function Selection

Choosing the loss function is the single most consequential decision you will make when training any AI model. It is the mathematical statement of what counts as a good prediction — and every model you will ever build, from a spam filter to a large language model, is trained by moving its internal settings to make the loss as small as possible. Understanding loss functions unlocks the ability to debug why a model is failing (it might be optimising the wrong thing), to compare models honestly, and to design custom objectives for novel tasks. This concept is the foundation for gradient descent, backpropagation, and every training loop you will see.

Imagine you are a teacher grading an exam. Each student's answer is a number (say, a forecast of tomorrow's temperature). The true temperature is 30°C. Student A predicts 29°C, student B predicts 20°C. You need a single number that says how bad each prediction is. Any sensible rule gives a higher number for B than for A. But there are many rules, and they differ in how much they punish an error. One rule might say 'the badness is the square of the error' — so A's error of 1 becomes 1, B's error of 10 becomes 100. Another rule might say 'the badness is just the size of the error' — so A gets 1 and B gets 10. The first rule punishes big errors far more harshly than small ones; the second treats all errors proportionally. This choice matters: if you tell a model 'minimise squared error,' it will bend over backwards to avoid any large error, even at the cost of many small ones. If you tell it 'minimise absolute error,' it will care equally about all errors, and might tolerate one big error if it reduces several small ones. The loss function is your instruction to the model about what you care about. Get this wrong, and the model will cheerfully do the wrong thing perfectly.

Think of choosing a loss function like choosing a measuring stick for a tailor making suits. Your customer wants a suit that fits. You can measure the fit in two ways: by the total length of cloth that hangs off (too big) or is missing (too tight), or by the square of that overhang. With a simple ruler (absolute error), a suit that is 2 cm too big in the shoulders and 2 cm too tight in the waist scores 4. With a squared ruler, each 2 cm deviation becomes 4, so the total is 8. Now imagine a suit that is 10 cm too big in one place and perfect everywhere else: the simple ruler scores 10, the squared ruler scores 100. The squared ruler makes that one glaring mistake look catastrophic — so a tailor told to minimise squared error will obsess over the worst-fitting part, even if it means making several other parts slightly off. The simple ruler treats all mistakes equally, so the tailor might accept one big mistake if it fixes many small ones. Neither is universally 'right' — it depends on what you, the customer, care about. If a single glaring error makes the suit unwearable, choose squared. If many small errors are as annoying as one big one, choose absolute. Real-world losses (like Huber) are compromises: they behave like squared error for small errors and like absolute error for large ones. The analogy breaks down because in AI, you don't just measure — you adjust thousands of parameters to reduce the loss, and the shape of the loss function (flat, steep, with bumps) determines how easy that adjustment is. A tailor can only adjust seams; a model adjusts every weight.

A loss function is a mathematical rule that takes a model's prediction and the true answer, and returns a single number — the cost — that says how wrong the prediction was; the model is trained by minimising this number.

You have already mastered the concept of 'activation functions' — the nonlinearity that gives a neuron its output. The loss function sits right after the activation: the activation produces the prediction, and the loss measures how far that prediction is from the truth. You have also seen 'gradient descent' — the process of nudging weights to reduce the loss. Without a loss function, gradient descent has nothing to aim at. In your library notes, you have 'Deep Learning From Scratch with Python (O'Reilly ed.)', which says: 'Next-token prediction is multi-class classification over the vocabulary, which is why cross-entropy is the natural loss.' This page builds the foundation for that claim — you will see why cross-entropy suits classification, and why squared error would be a poor choice for predicting which word comes next. Your notes on 'adaptive filtering' use squared error as the default loss; you will see here why that is a reasonable default for signal tracking, and when it fails.