Loss Function Selection

Choosing how to measure a mistake is the most important design choice in building an AI model because it dictates what the model prioritizes. Every model learns by trying to reduce this 'cost' score. If you define the cost incorrectly, the model will become very good at optimizing the wrong goal. Understanding this allows you to fix models that are failing in specific ways—for example, a model that is afraid to make big mistakes versus one that ignores small errors. This concept is the foundation for how any learning system improves over time.

Imagine you are a teacher grading an exam where each student's answer is a number, such as a forecast of tomorrow's temperature. The true temperature is 30°C. Student A predicts 29°C, making an error of 1 degree. Student B predicts 20°C, making an error of 10 degrees. You need a single number that says how bad each prediction is. Any sensible rule gives a higher number for B than for A, but the rules differ in how much they punish the error.

Think of choosing a loss function like choosing a measuring stick for a tailor making suits. Your customer wants a suit that fits. You can measure the fit in two ways: by the total length of cloth that hangs off (too big) or is missing (too tight), or by the square of that overhang. With a simple ruler (absolute error), a suit that is 2 cm too big in the shoulders and 2 cm too tight in the waist scores 4. With a squared ruler, each 2 cm deviation becomes 4, so the total is 8. Now imagine a suit that is 10 cm too big in one place and perfect everywhere else: the simple ruler scores 10, the squared ruler scores 100. The squared ruler makes that one glaring mistake look catastrophic — so a tailor told to minimise squared error will obsess over the worst-fitting part, even if it means making several other parts slightly off. The simple ruler treats all mistakes equally, so the tailor might accept one big mistake if it fixes many small ones. Neither is universally 'right' — it depends on what you, the customer, care about. If a single glaring error makes the suit unwearable, choose squared. If many small errors are as annoying as one big one, choose absolute. Real-world losses (like Huber) are compromises: they behave like squared error for small errors and like absolute error for large ones. The analogy breaks down because in AI, you don't just measure — you adjust thousands of parameters to reduce the loss, and the shape of the loss function (flat, steep, with bumps) determines how easy that adjustment is. A tailor can only adjust seams; a model adjusts every weight.

A loss function is a mathematical rule that takes a model's prediction and the true answer, and returns a single number — the cost — that says how wrong the prediction was; the model is trained by minimising this number.

You have already mastered the concept of 'activation functions' — the nonlinearity that gives a neuron its output. The loss function sits right after the activation: the activation produces the prediction, and the loss measures how far that prediction is from the truth. You have also seen 'gradient descent' — the process of nudging weights to reduce the loss. Without a loss function, gradient descent has nothing to aim at. In your library notes, you have 'Deep Learning From Scratch with Python (O'Reilly ed.)', which says: 'Next-token prediction is multi-class classification over the vocabulary, which is why cross-entropy is the natural loss.' This page builds the foundation for that claim — you will see why cross-entropy suits classification, and why squared error would be a poor choice for predicting which word comes next. Your notes on 'adaptive filtering' use squared error as the default loss; you will see here why that is a reasonable default for signal tracking, and when it fails.