Linear Regression

Linear regression is the first model you will learn, and it unlocks every other. It is the foundational model of applied statistics and the default starting point for almost any prediction problem. Master this and you can: predict house prices, forecast sales, estimate risk, and interpret which factors actually matter — because the coefficients are directly readable as marginal effects. It unlocks further study: this is the prerequisite for Logistic Regression (predicting yes/no), Polynomial Regression (curved relationships), and the entire train-validate-test discipline of Machine Learning Foundations. Without linear regression, every later model is an incantation; with it, you see the arithmetic underneath.

Imagine you want to predict how much a house will sell for. You know its size in square feet, its number of bedrooms, and its age. You have a list of past sales, with all those numbers for each sold house. Linear regression is the answer to: 'What is the pattern that connects these inputs to the price?' The pattern is assumed to be a straight-line relationship — not a curve, not a step, but a flat, steady rise or fall. For a single input, say size, the model draws a straight line through a scatter plot of past sales, and uses that line to read off the price for a new size. For many inputs, the line becomes a plane or a hyperplane (a flat shape in many dimensions), but the idea is the same: find the flat shape that comes as close as possible to all the observed points. 'As close as possible' is defined precisely: the sum of the squared distances from the line to each observed point, measured vertically (the error in the price you are predicting). Squaring makes large errors weigh far more than small ones. Then the work of linear regression is finding the line that makes that sum as small as it can be. That process — minimising the sum of squared errors — is the whole algorithm: a search over all possible slopes and intercepts for the one that makes the line sit best among the data.

Think of a diamond merchant weighing raw stones. She has a simple scale: a long rod balanced on a pivot. To weigh a stone, she places it on one end and adds standard weights to the other until the rod is level. The number of standard weights that balances the stone is its weight. Linear regression is a more sophisticated version of that scale. Instead of a single pan, imagine a rod with several pans hanging along its length. Each pan holds a different factor: one pan for floor area, one for number of bedrooms, one for age. And each pan hangs from the rod at a different distance from the pivot — that distance is the coefficient, the 'weight' that factor gets. To predict a price, you load each pan with that house's value and see how far the rod tilts. The total tilt is the predicted price. The art is choosing those distances along the rod (the coefficients) so that, across hundreds of past stones, the rod predicts their known weights as closely as possible. That is training. The analogy stops working where the merchant's scale has a fixed geometry — lengths are there to balance, but in regression those lengths are not given; they are learned from data. Also, a scale is physical and perfectly deterministic, while regression assumes the world is approximately flat but noisy — the points rarely lie exactly on the rod, only near it. The merchant can make her scale perfectly; the statistician can only make it best on average.

Linear regression is a way of finding the straight-line (or flat higher-dimensional) relationship between one or more input numbers and a continuous output number, by choosing the line that minimises the sum of the squared differences between its predictions and the actual observed outputs.

This is the first model in your Machine Learning Foundations path. You have not yet learned any others, but this one is the bedrock. Once you have it, your neighbouring study topics — Logistic Regression, Polynomial Regression, Decision Trees, K-Nearest Neighbours, Naive Bayes — each stand as alternatives to this. The key insight from your library: 'Every principal component is a linear combination of the original features — interpretability is traded for compression.' That is a direct descendant of today's idea: a linear combination means adding weighted inputs together, exactly what you will do here. Logistic Regression, which you will study next, is a direct extension: it takes the same linear combination but squashes it between 0 and 1 to predict a probability. Polynomial Regression is linear regression with squared or cubed input terms, but the model is still 'linear in the parameters' — meaning the coefficients are still combined by addition and multiplication. So today you are not just learning one method; you are learning the grammar of almost every model to come.