Hyperparameter Tuning

This unlocks how machine learning models are practically made to work. Without tuning, a model you train on past loan data, product images, or text will perform far below its potential — or fail to learn at all. Tuning is what turns a model that 'kind of works' into one that is reliable enough to ship. Mastering this lets you understand why a model that worked in a tutorial fails on your data, how to fix it, and why training runs in industry cost millions. It also connects forward to optimisation, gradient descent, and regularisation — the machinery under every large language model you use.

Imagine you are a chef developing a new recipe. The ingredients are your model's settings: the learning rate (how aggressively the model adjusts), the depth (how many layers it has), the regularisation strength (how strictly you prevent it from memorising instead of understanding). Each setting is a knob you can turn before training begins. Once training starts, the model does not turn these knobs itself — it only adjusts its own weights based on the setting you chose. So if you set the learning rate too high, the model jumps around and never settles; too low, and it crawls so slowly that training takes forever. Depth too large and the model memorises the training data but fails on new data; too small and it cannot capture the pattern at all. Tuning is the systematic process of searching over combinations of these knobs to find the one that makes the model perform best on data it has never seen.

Think of tuning hyperparameters like seasoning a dish of biryani. The rice, the spices, the meat — those are the fixed ingredients, like the model architecture. But the amount of salt, the heat level, the cooking time — those are the hyperparameters. You do not know the right amount of salt without tasting. You could try every possible amount (grid search), but that is wasteful. Instead, a good cook tastes, adjusts, tastes again, and uses experience to guess the next adjustment. In machine learning, Bayesian optimisation does exactly this: it tastes (evaluates the model), learns from the taste (fits a surrogate model), and chooses the next amount to try. Where this analogy breaks down: a cook can taste instantly, but each model evaluation takes minutes or hours. Also, a cook knows the dish 'should' taste salty enough without every possible test; model evaluators have no such prior, so they rely on statistical search rather than intuition alone.

Hyperparameter tuning is the process of searching over the settings that are not learned from data — like learning rate, model depth, and regularisation strength — using grid search, random search, or model-based search, to find the combination that gives the best performance on validation data.

You have not yet met gradient descent, but tuning is inseparable from it: the learning rate is the single most consequential hyperparameter, and schedules that change it during training are tuning decisions. Also connected is regularisation — choosing its strength is itself a hyperparameter. And the search methods here (random, grid, Bayesian) are the same search ideas used across optimisation, where the goal is to find the best input without knowing the landscape in advance. Your library notes on convex optimisation remind us that deep learning is not convex, so tuning is empirical rather than guaranteed — that is precisely why grids and random trials matter.