Double Descent

This concept is the reason modern AI models are enormous. It explains why adding more parameters to a neural network can improve test performance even after the model has memorized the training data. If you are going to work with large language models, dense recommender systems, or any deep learning model, you will hear about 'overparameterized' models that should overfit but don't. Double descent is the key to understanding why. It connects to your notes on Gradient Descent and Stochastic Gradient Descent (the optimizer's implicit regularization is part of the story) and to Hyperparameter Tuning (the size of the model is the most important knob you can turn). Master this and you'll see why the textbook advice 'smaller models are safer' can be exactly wrong.

You know the classical story: a model with too little capacity underfits, a model with too much capacity overfits, and there's a sweet spot in the middle. You've seen the U-shaped curve: error high on the left, falling to a minimum, then rising again on the right. Double descent says that for modern models—deep neural networks, huge linear models—the curve is not a U. It falls, then rises to a peak, then falls again as capacity keeps growing. The peak is not where overfitting is worst. It is where the model has just enough capacity to fit the training data perfectly—the interpolation threshold. Past that point, the model continues to fit the training data perfectly (training error is zero) but test error drops again, sometimes below the earlier minimum. The intuition is that once a model has enough parameters to memorize the data, the optimizer can find a solution that is both a perfect fit and smooth—a solution that interpolates the data but does not wiggle. Classical simplicity arguments said such a solution was impossible; empirical results say it is not only possible but typical. The reason is that the optimizer itself, through gradient descent, implicitly favors smooth solutions. The noise in stochastic gradient descent, the learning rate schedule, and the choices of initialization all act as a regularizer that is not in the loss function but is in the training procedure.

Think of a chef learning to reproduce a set of recipes. The recipes are the training data—ten distinct dishes. The chef's capacity is the number of ingredients available. With only three ingredients, the chef cannot make all ten dishes correctly; that is underfitting. As the chef gains ingredients, the dishes start to resemble the originals, and the error falls. At around ten ingredients, the chef can exactly reproduce each recipe—that is the interpolation threshold. A classical chef would stop here, fearing that having more ingredients would cause the chef to memorize the recipes, only able to cook the exact dishes and unable to cook anything else. Indeed, as the chef gains ingredients beyond ten, the classical chef does start to overfit: the dishes become bizarre, tailored to the exact measurements of the training data, and the chef cannot handle a new variation. That is the rising part of the U. But double descent says something strange happens if the chef keeps gaining ingredients. At, say, a hundred ingredients, the chef does not overfit. The chef has so many ingredients that there are infinitely many ways to reproduce the exact recipes. Among those many ways, the training process (the chef's apprenticeship) naturally selects a way that uses the ingredients smoothly, not erratically. The resulting dishes are not only exact copies of the training dishes but also robust to small variations—they generalize. The analogy breaks down because a chef's capacity is not just the number of ingredients; a real chef also learns techniques and has a mental model. But the key point stands: more capacity does not inevitibly mean overfitting. It is only after the model crosses the interpolation threshold that a second regime emerges, one where the model can fit the data in many ways, and the training dynamics pick a good one.

Double descent is the empirical observation that the test error of a model, as a function of its capacity (such as the number of parameters), falls, then rises to a peak at the point where the model can just fit the training data exactly, and then falls again as capacity increases further, in contrast to the classical U-shaped curve that predicts test error rising monotonically after a certain point.

You have notes on Gradient Descent and Stochastic Gradient Descent, Convex Optimization, Hyperparameter Tuning, and more. Double descent is a counterexample to the classical bias-variance tradeoff that underlies those notes. It sits directly in the story of Optimization: it is a property of the loss surface and the optimizer's path on it. The implicit regularization from the optimizer is part of the explanation—this connects to your notes on Stochastic Gradient Descent, where you noted that gradient noise is not purely a cost but can help generalization. It also connects to Convex Optimization: classical double descent was first studied in linear models, which are convex, so the phenomenon is not a consequence of nonconvexity. It is also a reason why hyperparameter tuning for model size can behave counterintuitively: increasing the size of a model can decrease test error, so tuning only over small sizes misses the best model. Your notes on Inverse Problems and Regularization are also relevant because the implicit regularization is a form of regularization that is not part of the loss.