Normal Equations

This concept is the mathematical core of 'least-squares fitting', which is how almost every AI system learns from data. When a model predicts a number (house price, probability of fraud, next word in a sentence), it is solving a least-squares problem — and the normal equations are the closed-form, exact solution for small- to medium-sized data. This unlocks further study in: Gradient Descent (the method used when data is too big for the normal equations), Overfitting and Regularization, and Ridge Regression. Understanding the normal equations gives you a concrete anchor: you will meet its geometric idea — the residual must be perpendicular to what you can explain — again in PCA, in projections, and in the linear layers of neural networks.

Imagine you have a bunch of data points scattered on a graph — for instance, the more hours you study, the higher your exam score. You want to draw a straight line through these points. The line won't pass through every point, but it should come as close as possible to all of them. 'Least squares' means you choose the line that minimizes the sum of the squared vertical distances from the data points to the line. Why squared? Because squaring makes big errors count much more than small ones, so the line is pulled toward the points that are far away — you care more about avoiding huge mistakes than tiny ones. The normal equations are a way to find the perfect slope and intercept of that line without guessing or trial and error. They are 'normal' because they come from a geometric condition: the errors (the vertical gaps) must be perpendicular to the direction of the line. That might sound abstract, but it is just the mathematical way of saying 'the line is as close as it can possibly be'. The normal equations are like a recipe: you take the data, multiply it in a certain way, and the answer pops out. That multiplication involves a matrix — a table of numbers — and in the next section we will learn exactly what that multiplication is, step by step.

Think of a real estate agent trying to guess the price of a house from its size. She has a list of past sales: each house has a size (in square feet) and a known price. She wants to find the 'best' formula: price = (size × weight) + base. You can imagine her drawing a straight line through a scatter plot of size vs price. But which line? She could try many lines, but she wants the one that is 'most fair' — the one where, on average, her guess is as close as possible to the real price. She decides to minimize the total of the squared errors (the squared difference between her guess and the actual price). Now, here is the key insight: the 'errors' (the vertical gaps between her line and the data) should not have any upward or downward pattern. If all the errors were positive (her guesses too low), she could lower the line to fix that. If all were negative, she could raise it. The only time she cannot improve is when the errors are balanced — in a sense, they are perpendicular to the line. That perpendicularity is what the normal equations enforce. This analogy works for one variable (size), but it extends to many variables (size, location, age) — the 'line' becomes a 'hyperplane', and the geometry stays the same. Where the analogy stops: the real estate agent might want to weigh different factors differently, and sometimes a straight line is not enough (a curved relationship). But for now, a straight line is exactly what the normal equations find.

The normal equations are the linear system whose solution is the least-squares fit; they are derived from the geometric condition that the residual vector is orthogonal to the column space of the design matrix.

You have notes on Orthogonality and Projections, Matrix Multiplication, and Linear Transformations. The normal equations are the direct child of those ideas: the least-squares solution is a projection of the target vector onto the column space of the design matrix, and the residual being perpendicular is the geometric content. This is a key point in your Linear Algebra for Machine Learning Workbook. It also connects to Determinants and Inverses (solving the system often uses matrix inversion), Eigenvalues (the condition number of a matrix, which the normal equations make worse), and later to Gradient Descent (an iterative alternative).