Partial Least Squares

This unlocks the next stage of your journey: applying regression when your predictors are many, overlapping, and noisy. With Partial Least Squares you will be able to build models for real-world problems like predicting crop yield from hundreds of soil measurements, or forecasting sales from dozens of overlapping marketing channels — situations where ordinary regression collapses. You will also understand what 'supervised dimensionality reduction' means, and how choosing the right number of components is the single most important tuning decision. From here you can move on to more advanced topics like regularised regression, cross-validated model selection, and the full machinery of modern statistical learning.

Imagine you have a big table of numbers — say, measurements from 50 sensors on a machine, and you want to predict how many hours the machine will run before it fails. Each sensor is a 'predictor', and the failure time is your 'response'. If you tried to use all 50 sensors in a plain regression, the model would be a mess: many sensors are measuring the same underlying thing (temperature, vibration, pressure), so they are 'collinear' — they move together. The model can't tell which one matters, and its predictions become unstable: a tiny change in the data flips the answer. What you really want is to find the few underlying 'forces' — maybe one is 'overall heat stress', another is 'mechanical wear' — that drive both the sensors and the failure time. Partial Least Squares (PLS) does exactly that. It looks at the sensors and the failure times together, and pulls out a small number of 'components' — each one a weighted combination of the sensors — that explain both the variation in the sensors and, crucially, the variation in failure time. It's like a detective who, instead of interviewing every witness separately (and getting confused by contradictions), groups the witnesses by what they actually saw, and uses each group's consensus to solve the crime. The key difference from a method called PCA (Principal Component Analysis) is that PCA only looks at the sensors — it finds directions of high variance in the sensors alone, without caring about the failure time. PLS is 'supervised': it lets the response guide which components to build, so the components are the ones that actually predict well. In practice, you end up with a model that uses just a handful of components instead of 50 predictors, is more stable, and often predicts better because it ignores the noise.

Think of a group of tourists trying to find the best restaurant in a city. They have a map with many landmarks (predictors), but they don't know which landmarks matter. PLS is like a savvy local guide. The guide doesn't just look at the map and pick the most prominent landmarks (that would be PCA — it might pick a big shopping mall that has nothing to do with food). Instead, the guide watches where the locals actually go (the response). The guide notices that whenever locals go to a certain neighborhood, they end up at good restaurants. So the guide combines several landmarks — the presence of a park, a metro station, and a bakery — into a single 'restaurant-friendliness' score (a component). This score is built specifically to predict where the locals go. The guide repeats this a few times, each time finding a new combination of landmarks that explains the remaining 'food-seeking' behavior. At the end, the guide has a handful of scores — not 50 separate landmarks — that predict restaurant quality well. Where the analogy breaks down: the guide is a human with intuition; PLS is a mathematical algorithm that finds these combinations automatically by maximising a specific numerical goal (covariance). Also, the guide can see all the landmarks at once; PLS builds components one at a time, each new one accounting for what the previous ones missed. And crucially, the guide's judgment is subjective; PLS's components are deterministic — given the same data, you always get the same components.

Partial Least Squares (PLS) is a supervised regression method that projects the predictors and the response into a lower-dimensional space by constructing components that maximise the covariance between them, and then fits a linear regression on those components.

You haven't met PCA yet, but you have met the idea of correlation and scatterplots — how two variables move together. PLS builds on that by looking at the joint variation between many predictors and one response. It's related to ANOVA in the sense of partitioning variance, but here we partition the variance of predictors *and* response together. And it connects directly to Cross-Validation — you'll use cross-validation to choose how many components to keep, because using too many overfits and too few underfits. Think of PLS as the supervised cousin of PCA, which you'll learn later; both reduce dimensions, but PLS uses the response to guide the reduction.