Nonsmooth Optimization

This is the gateway to solving real-world optimisation problems where the objective is not a smooth curve. Without it, you are stuck: gradient descent will stall or jump at kinks, and you cannot train models with L1 penalties (which are everywhere in sparse models and feature selection) or hinge losses (used in support vector machines). Mastering this unlocks further study in Proximal Gradient Methods, which power modern compressed sensing and image reconstruction, and gives you the tools to handle non-differentiable objectives in your own applied work. It also deepens your understanding of Convex Optimization, as nonsmoothness is the main reason a problem stops being 'solved' by simple calculus.

Imagine you are trying to find the lowest point in a mountain range, but you can only walk by feeling the ground under your feet. If the ground is smooth, you can always tell which direction goes downhill. But what if you hit a ridge or a sharp crease in the rock? At that exact line, the ground is not smooth—it changes direction abruptly. You cannot say 'the slope here is exactly this' because the slope is different on each side. You can still make progress: you know that any direction that goes downhill is fine, even if there is no single 'steepest' direction. This is the core of nonsmooth optimisation: when the objective has sharp corners (kinks) or jumps, you cannot rely on the usual concept of a slope (the derivative), because it simply does not exist at those points. Instead, you use a generalised notion—the subgradient—which is any direction that falls no slower than the function itself. You then step in a direction that, while not uniquely 'steepest', is guaranteed to reduce the objective. This is what allows optimisation to continue even at the kinks. In machine learning, these kinks appear all the time: the L1 penalty (which is the sum of absolute values) has a kink at zero, and the hinge loss (used in SVMs) has a kink where the margin is exactly met. Ignoring these kinks means your algorithm gets stuck or misbehaves exactly at the points that matter most.

Think of a delivery driver navigating a city grid with a long rectangular block of buildings in the centre. The driver's goal is to find the fastest route (minimise travel time). On the smooth streets, the driver can tell exactly which turn is 'downhill' in terms of time. But the central block has a sharp corner—the intersection of two one-way streets. At that corner, the 'slope' of time is not defined: coming from one direction, the best way is to turn left; coming from another, it is to turn right. The driver does not stop and panic; instead, they pick any direction that is not worse. In optimisation terms, the driver uses a 'subgradient'—any direction that does not increase the objective (or at least, does so at a rate no worse than linear). The driver also uses a trick: they can 'clip' or 'round' the corner (like the Huber function) to make it smooth enough to navigate with normal methods, accepting a small trade-off in precision. But this analogy breaks down when the kinks are not physical corners but mathematical discontinuities—like when the objective jumps from one value to another with a gap, such as a step function. No driver can handle a jump, and neither can a plain gradient method; you need specialised operators that look at the function from both sides. So the analogy holds for the common case of kinks, but not for jumps—those require a different toolset.

Nonsmooth optimisation is the process of minimising an objective function that has points where the derivative does not exist (kinks or discontinuities), requiring generalised notions like subgradients or smoothing techniques to make progress.

You have studied Gradient Descent, where the algorithm relies on the smooth slope (derivative) to step downhill. That works beautifully when the loss is smooth, like a quadratic. But your library notes on Convex Optimization and Proximal Gradient Methods hint at the fact that many real-world objectives are not smooth. This concept fills that gap: it tells you what to do when the derivative does not exist. It also connects to your notes on Least Squares Regression: the plain least-squares loss is smooth, but when you add an L1 penalty (as in LASSO), the objective becomes nonsmooth at the point where any coefficient is exactly zero, which is exactly where sparsity happens. So this is the mathematical foundation for why L1 regularisation creates zeros in a way that L2 (smooth) does not. It also relates to your notes on Duality and Lagrange Multipliers, as nonsmoothness often appears when constraints are folded into the objective as penalties (e.g., L1 is a nonsmooth penalty).