Least Squares Regression

This is the foundation of fitting any line or curve to data — the method behind price prediction, demand forecasting, and every regression model you'll meet. Mastering this unlocks gradient descent (the engine of all neural network training), regularization (how we stop models from overfitting), and the entire idea of optimization: finding the best parameters by minimizing an objective. You'll never write a machine learning model that doesn't, at its heart, do what you'll learn here: choose numbers that make predictions as close to reality as possible.

You have a scatter of data points — say, house prices versus square footage. You want a line that goes through them as best as possible. But 'best' is vague. Least squares makes it precise: we measure the vertical distance from each point to the line (that's the 'error' or 'residual'), square it, and add them up. The line with the smallest total of squared errors is the best fit. Why square? Because squaring makes every error positive (so they don't cancel out), and it punishes large errors much more than small ones — if one point is ten units away, it contributes 100 to the total, whereas two points each five units away contribute only 50. The squaring also gives us a smooth, bowl-shaped curve that has a clear bottom — the point with the minimum sum is the line we want.

Imagine you're the captain of a ship and you have a map with a set of targets you need to hit as closely as possible. You can't hit all of them exactly, so you want to find a single straight course that gets you as close to all targets as possible. Each target is a data point. You decide to measure your miss distance as the square of how far you are from each target (so a big miss is much worse than two small ones). The course you choose is the one that minimizes the sum of squared miss distances. This works well when targets are roughly in a line. But it breaks down if one target is an outlier — say, an old lighthouse that's misplaced. That one far-away target will pull your whole course toward it, because its squared distance dominates everything else. This explains why least squares is sensitive to outliers, and why we sometimes switch to a 'robust' method that doesn't square so aggressively (like the Huber function).

Least squares regression is a method for finding the line (or more generally, a curve) that fits a set of data points by choosing parameters (like slope and intercept) that minimize the sum of the squared vertical distances between each point and the line.

You haven't studied any other topics yet, so this stands alone. But it is the seed of everything: it introduces the idea of an 'objective function' (the sum of squared errors) and 'parameters' (slope and intercept) that we adjust. The same structure appears in gradient descent (where we step parameters downhill) and convex optimization (where the sum of squared errors has a single best answer). This is also the first time you'll see a 'closed-form solution' — a direct formula that gives the best parameters without iteration. Later, when we learn about regularization and robust losses, you'll understand exactly what they're fixing: the limitations of that squared error term.