Proximal Gradient Methods

You are learning this because it is the standard way to train models that have a kink in their loss — most importantly L1 regularisation, which is how you get sparse models (models that ignore most of your inputs). Once you have this, you can read any lasso or sparse-model paper without flinching. It also gives you a clean mental model for why your framework can optimise things like 'loss + regularisation' as one step — because it is literally one step, computed by this method. This will unlock further study in optimisation: it is the bridge between the gradient descent you already know and the proximal and nonsmooth methods your library lists as leading to Deep Learning, Reinforcement Learning and Regularization.

Suppose you are trying to reach the lowest point in a valley, but the valley floor has a deep crack running through it — you can't stand exactly on the crack, and the slope on either side is different. Ordinary gradient descent says: look at the slope where you are, step downhill. But if you step down to the crack, you can't stop there — you are forced to land on one side or the other. Proximal gradient methods split the problem: first, take a regular downhill step as if the crack weren't there; then, in a second step, adjust the result to respect the crack — pushing the value just enough to the nearest allowed side. That second adjustment is the 'proximal operator'. The trick is that for common penalties like L1 (which creates the crack), that adjustment has a simple closed form — a single arithmetic operation you can do by hand. So the whole method is just: take a gradient step, then apply a simple 'shrink towards zero' rule, repeat. Each step stays cheap, and the result respects the nonsmooth structure of the problem.

Think of a hiker trying to reach the lowest point of a valley, but the valley has a deep crevice running along the bottom — the hiker cannot stand inside the crevice; they can only stand on either edge. Ordinary gradient descent would tell them to step downhill, but if the step lands them in the crevice, they are stuck (the slope is undefined there). The split strategy says: first, take a bold downhill step as if the crevice weren't there (that's the gradient step). Then, look at where you landed; if you're in the crevice, the proximal operator pushes you to the nearest edge you can stand on — not arbitrarily, but by a specific, simple rule. For L1 regularisation, that rule is: if the value is small, push it all the way to zero; if it's large, shrink it by a fixed amount. This 'shrinking' is called soft thresholding. The analogy breaks down: in a real landscape you can stand anywhere, but in optimisation the crevice represents a kink — a point where the slope doesn't exist — and the proximal operator is not a physical push but an algebraic formula that finds the closest allowed point. Also, the hiker doesn't have a map; they only feel the slope at their feet, which is why each step uses only local information.

Proximal gradient methods are an iterative optimisation technique that splits the objective into a smooth part (handled by a gradient step) and a nonsmooth part (handled by a proximal operator), where for common penalties like L1 the proximal operator has a closed-form expression, so each iteration is cheap and respects the kink.

You already have notes on Gradient Descent and Stochastic Gradient Descent. This is the next step: those methods assume the loss is smooth — you can always compute a slope. Proximal gradient methods drop that assumption. They are the natural extension when your loss has a kink, like an absolute value. Your library also ties this to Convex Optimization: when the problem is convex, proximal gradient methods have strong convergence guarantees. Your notes on Regularization mention L1 as the classic nonsmooth penalty — this is exactly how lasso is solved. It is also related to Inverse Problems, where regularisation is needed to make an ill-posed problem stable — proximal methods are the workhorse there. You do not need any new mathematics beyond what you will meet here: you need to know what a gradient step is (from your Gradient Descent notes) and what an absolute value is, and I will teach you the proximal operator from scratch.