Gaussian Process Regression

This unlocks the next stage of your statistical toolkit. You already know how to fit a line or a curve and get a single answer. Gaussian process regression gives you something different: every prediction comes with a range that says how confident the model is, and where it knows nothing it says so. This is the machinery behind Bayesian optimisation (used to tune hyperparameters of expensive models), behind the uncertainty estimates in active learning (deciding which data to label next), and behind many modern approaches to safe reinforcement learning where the model must know what it does not know. It also leads naturally into the study of kernels, which appear again in support vector machines and in the attention mechanisms inside large language models.

You have a set of points, say houses sold with their prices. You want to predict the price of a new house. A normal regression model would give you one number. But you also want to know: how sure am I? Near the points I have seen, I should be quite sure. Far away, I should be very unsure. Gaussian process regression does exactly this. It treats the entire function (the rule from house size to price) as an unknown that we observe through noisy data. Instead of finding one best function, it maintains a whole distribution of possible functions, all consistent with the data we have seen. When you ask for a prediction at a new point, it averages over all those plausible functions to give a mean, and it looks at how much they disagree at that point to give an uncertainty. Close to the data, all plausible functions agree, so the uncertainty is small. Far away, they diverge wildly, and the uncertainty grows. So you always get a prediction plus a measure of how much you should trust it.

Imagine you are a detective trying to reconstruct a suspect's daily routine from a few eyewitness reports. Each report is a data point: at 9 am they were at the café, at 2 pm at the office, at 7 pm at the gym. From these, you infer a plausible timeline. But you do not just want one timeline; you want to know, at 3 pm, where they probably were and how sure you are. The eyewitnesses are noisy, so multiple timelines fit the reports. Some timelines make them go straight from office to gym; others have them stop for coffee. Near the actual reports (around 9, 2, 7) all timelines agree. At 3:30 pm, they mostly agree roughly, but not exactly. At midnight, far from any report, you have no idea, and the spread across possible timelines is huge. Gaussian process regression is exactly this. The 'kernel' is your prior belief about how smooth a routine usually is — a smoother kernel says the suspect rarely teleports, so nearby times must have similar locations. The 'training points' are the eyewitness reports. The prediction at a new time is the average of all plausible timelines, and the uncertainty is how much they disagree. Where the analogy breaks: the detective might have additional physical constraints, like how fast a person can walk, or that they cannot be in two places at once. The Gaussian process only encodes smoothness through the kernel, not hard constraints like speed limits. It also assumes the underlying function is smooth in a very specific statistical sense, not that it has any particular shape (like linear or quadratic). So it is more flexible than a straight line, but it cannot know that a person cannot teleport unless you encode that in the kernel, which is hard to do precisely.

Gaussian process regression is a nonparametric Bayesian method that places a probability distribution over all possible functions consistent with the observed data, and uses that distribution to return, for every query point, a mean prediction and a calibrated measure of uncertainty.

You already have notes on cross-validation, which is about estimating how well a model will perform on new data. Gaussian process regression is a way of building a model that, by design, tells you its own uncertainty — so you can use it in cross-validation to see not just how well it does on average, but where it is confident and where it is not. You also have notes on scatterplots and relationships; a Gaussian process is what you get when you take the idea of fitting a smooth curve to a scatterplot and add a rigorous account of your uncertainty about that curve. It draws on the same statistical inference framework that you have been studying — the idea that we make conclusions from a sample with an explicit account of how wrong we might be. Gaussian process regression is the Bayesian version of that: it starts with a prior over functions and updates it with data to get a posterior over functions.