Least Squares Regression

You often need to find a single rule that describes a messy collection of numbers. Imagine you have a list of sales figures for different months and want to predict next month's sales. Or perhaps you are comparing the price of used cars against their age, looking for the general trend. Least squares regression is the standard tool for this. It finds the straight line (or curve) that sits closest to your data points, turning a cloud of scattered information into a clear, usable pattern. You use it whenever you want to say, 'Given these inputs, what is the most likely output?' without guessing wildly.

Let's look at a concrete example. Suppose you have three data points representing house prices (in tens of thousands of dollars) based on their size (in hundreds of square feet):
1. Size 20, Price $100
2. Size 40, Price $150
3. Size 60, Price $200

You want to draw a line through these. The line might not hit every point exactly. For each house, you measure the vertical gap between the actual price and the line's predicted price. This gap is called the error.

For point 1, if the line predicts $95, the error is 5 (100 - 95).
For point 2, if the line predicts $160, the error is -10 (150 - 160).
For point 3, if the line predicts $205, the error is -5 (200 - 205).

If you just added these errors (5 + (-10) + (-5)), you get 0. This is misleading because the line is clearly not a perfect fit; it misses points by significant amounts. To fix this, we square each error. Squaring makes every gap positive and exaggerates larger gaps.

Squared errors:
1. 5^2 = 25
2. (-10)^2 = 100
3. (-5)^2 = 25

Total squared error = 150.

The 'Least Squares' method searches for the specific line that makes this total sum as small as possible. Why square instead of just taking absolute values? Because squaring creates a smooth, bowl-shaped surface of errors. This shape has one clear bottom point, allowing us to calculate the exact best line using algebra, rather than having to guess and check endlessly.

Imagine you are holding a rubber band attached to a nail in a board. The nail represents your data point. You pull the rubber band taut so that it tries to sit as close to all nails as possible simultaneously. The tension in the rubber band represents the error. When you let go, the rubber band snaps into the position where the total tension (the sum of squared stretches) is minimized. This visualizes how the line settles into the 'valley' of lowest total error. Note that a single nail pulled far away (an outlier) will stretch the rubber band immensely, pulling the line toward it disproportionately strong.

Least squares regression is a technique for finding the best-fitting line or curve by choosing its settings—such as steepness and starting point—to minimize the sum of the squared vertical distances between each data point and the line.

In plain terms: It picks the equation that reduces the total amount of 'squaring-and-adding-up' error to its lowest possible value.

This concept is the bedrock for two adjacent ideas. First, gradient descent is a general method for finding the bottom of that bowl-shaped error surface by taking small steps downhill; least squares provides the specific shape that makes this process reliable and fast in many cases. Second, polynomial regression extends this idea to fit curves instead of straight lines, using the same principle of minimizing squared errors but allowing for more complex shapes.