Polynomial Regression

Polynomial regression is the first step beyond straight lines. Once you master it, you will understand why machine learning models can bend and curve to fit data—and why too much bending is a trap. This concept unlocks the study of overfitting, model evaluation, and the bias-variance tradeoff, which are the gates to every serious modelling technique that follows: splines, regularisation (ridge and lasso), and even the multi-layer neural networks behind modern AI.

So far, you have probably fit straight lines to data: something like 'for every extra year of experience, salary rises by ₹50,000.' But the world is not always straight. Prices rise, then fall. Growth accelerates, then slows. A straight line cannot capture a U-shape or an S-curve. Polynomial regression lets the line bend by adding powers of the input: the input squared, cubed, and so on. Instead of drawing just y = a + b*x, we draw y = a + b*x + c*x² + d*x³. Each power adds one curve. The first power is the straight trend, the second power adds a single bend (like a valley or a hill), the third adds an S-curve, and higher powers add more wiggles. With enough powers, you can draw nearly any shape—but there is a price: a curve that wiggles too much will fit the training data perfectly and fail on new data. That is the central tension of this concept: flexibility versus reliability.

Imagine you are tracing the outline of a leaf with a pencil. A straight line from the base to the tip would capture the general direction but none of the leaf's curves. Your first attempt uses one straight line: it is simple, but it misses most of the shape. So you try a curved stroke—the leaf's outline bends gently, and you get closer. But the leaf has wiggles, so you keep adding gentle curves to your stroke: a little bend here, a small wave there. Now the outline follows the leaf almost perfectly. However, if the leaf has a dirt speck or a torn edge, your pencil will faithfully trace that tiny imperfection too. If you then try to draw a different leaf from the same species, your overly detailed outline will fail because it has copied the speck of the first leaf, not the general leaf shape. Polynomial regression works exactly this way: adding powers is like adding curves to your pencil stroke. Low degree: straight line, too simple. Medium degree: captures the main bends, works well on new leaves. Very high degree: traces every speck, fails on anything new.

Polynomial regression is an extension of linear regression that predicts an outcome using powers of the input (like x², x³) as extra predictors, so the fitted curve can bend, while still being a linear model in the parameters (the coefficients being learned).

This concept builds directly on Linear Regression, which you have already studied. In linear regression, you fitted a straight line: y = a + b*x. Here, you will do the same arithmetic but with extra columns for x², x³, etc. The parameters (a, b, c, ...) are still found by the same least-squares method. The change is purely in the input features, not in the learning algorithm. This also connects to your notes on Ridge and Lasso Regression, which are the standard ways to control the wild bends that high-degree polynomials introduce. If you go on to study Splines (mentioned in your notes), you will see that they achieve the same flexibility as high-degree polynomials but with much better stability.