Normal Equations

Imagine you are trying to predict next month’s electricity bill based on last month’s usage. You have data: for ten months, you recorded how much power was used (in kilowatt-hours) and what the bill actually was. You suspect a simple relationship: the bill is roughly proportional to usage. You want to find the single best multiplier that connects usage to cost. The method known as the normal equations gives you a direct recipe to calculate that multiplier from your data table. They are worth learning because they provide an exact answer without needing to guess or iterate, which is helpful when your dataset is small enough to fit on one screen and you need a clear, transparent calculation rather than a black-box approximation.

Let’s work through a concrete example. Suppose you have only two data points: Month 1 had 100 kWh usage and a $50 bill. Month 2 had 200 kWh usage and a $100 bill. You want to fit the line Cost = m * Usage.

If you pick m = 0.4, your predictions are $40 (for 100 kWh) and $80 (for 200 kWh). The errors (the difference between predicted and actual) are -$10 and -$20. You want the error to be as small as possible in total. Specifically, you minimize the sum of the squared errors: (-10)^2 + (-20)^2 = 100 + 400 = 500.

Now try m = 0.6. Predictions are $60 and $120. Errors are +$10 and +$20. Sum of squares: 100 + 400 = 500 again.

Try m = 0.5. Predictions are $50 and $100. Errors are $0 and $0. Sum of squares is 0. This is clearly better.

The normal equations tell you how to find that perfect m without guessing. The formula for these equations, when applied to finding a single multiplier like this, effectively balances the data so that the calculation accounts for every piece of information equally. They work by ensuring that the remaining error (the part of the bill your simple line couldn't explain, often called the residual) has no connection to the input data (usage). In geometric terms, if you draw a vector representing all your actual bills, and another vector representing all your predicted bills, the gap between them must stand at a right angle to the space where your predictions live. If the gap weren’t perpendicular, you could tilt your line slightly to reduce the error further. The normal equations calculate exactly that tilting direction so the gap becomes perpendicular.

A key part of this process is understanding projection. When we say the prediction is 'projected' onto the space of possible usage values, we mean we are finding the closest possible match within our model's capabilities to the real data. Think of it like shining a light straight down on the usage line; the shadow cast by the actual bill amounts onto that line is your prediction.

Think of holding a string attached to a nail on a wall. The nail represents your actual data point (the true bill). You pull the other end of the string toward a straight ruler laid out on the floor beneath the nail. The ruler represents all possible predictions your model can make (since it's a simple line, it's a 1D space). When you let go, the string tightens and becomes perpendicular to the ruler. The point where the string touches the ruler is your 'best fit' prediction. The length of the remaining slack in the string is your error. If the string were at an angle to the ruler, you could slide your hand along the ruler to make the slack shorter. But when it’s perpendicular, any move along the ruler would only increase the length of the slack. That perpendicular point is exactly what the normal equations find.

The normal equations are a specific set of calculations, represented by the formula (X^T X) beta = X^T y, which solve for the best-fit parameters by forcing the prediction error to be orthogonal (at right angles) to the input data features.

A residual is the difference between an observed value and the value predicted by the model.

To project a value onto a space is to find its closest corresponding point within that space.

This builds directly on your understanding of Matrix Multiplication and Linear Transformations, as these tools allow you to calculate how inputs map to predictions. It also relates to Orthogonality and Projections, because the 'perpendicular' condition means you are projecting the actual outcomes onto the space of possible predictions.