Gradient Descent

Gradient descent is the single algorithm that makes modern AI possible. Every large language model, every recommendation system, and every vision model is trained by adjusting its internal numbers — often billions of them — using gradient descent. Without it, a neural network would be a lifeless pile of random numbers. Mastering this concept unlocks the rest of machine learning: you will understand how models learn, how training works, and why choosing the right step size can make or break a training run. It is also the foundation for every optimization technique you will meet later — stochastic gradient descent, momentum, adaptive methods like Adam — all of which are just gradient descent with tweaks. Even the production systems you will build — fine-tuning an LLM, training a fraud detector — are gradient descent at heart.

Imagine you are standing on a foggy, uneven valley and you want to reach the lowest point — the bottom of the valley — but you cannot see far ahead. You can only feel the slope of the ground under your feet. The rule is simple: take a step downhill, in the direction where the ground descends most steeply. Then feel the slope again, and take another step. Repeat. This is gradient descent. It is called 'descent' because you are going down, and 'gradient' because that is the mathematical name for the direction and steepness of the slope at your feet. The size of each step you take is the learning rate — if you take huge strides, you might overshoot the bottom and end up on the other side, even higher than before. If you take tiny shuffles, you will get there eventually, but it will take forever. The trick is to pick a step size that gets you down reasonably fast without overshooting. In machine learning, the valley is not made of land — it is the loss surface, a mathematical landscape where the height at any point tells you how badly the model is performing. Your position is the current set of weights (the numbers inside the model), and your goal is to find the set of weights that makes the loss as small as possible. Gradient descent is the walk you take to get there.

The hiker in fog — this is the classic analogy for gradient descent, and it works well because it captures every essential ingredient. A hiker wants to descend a hill to reach the valley floor, but the fog is thick: they can only see a few metres in front of them. They cannot see the whole landscape, so they cannot plan a perfect path. Instead, they use local information: they feel the slope under their boots and step in the direction of steepest descent. That is the gradient. They also decide how big a step to take — that is the learning rate. Take steps that are too big, and they might leap over a small gully and land on a slope that leads right back up, or even miss the valley entirely and wander onto another hill. Take steps that are too small, and they will eventually reach the valley, but they might be out there for days. The fog also hides something else: the valley floor might not be the only low point. There could be a small dip, a local minimum, that looks like the bottom from up close, but is actually much higher than the true valley just beyond a ridge. The hiker, following the slope, will happily settle there, thinking they have reached the goal. In machine learning, this is the problem of local optima — gradient descent can get stuck in a shallow dip and never find the truly best weights. But here is the surprise: in deep learning, with billions of weights, researchers have found that local minima are often almost as good as the global one, and getting stuck is rarely the disaster it sounds like. So the hiker analogy holds — the fog, the steps, the local dips — but the ending is friendlier than you might fear.

Gradient descent is an iterative optimization algorithm that repeatedly adjusts a set of parameters by moving them a small step in the direction opposite to the gradient of a loss function, aiming to find the parameter values that make the loss as small as possible.

You have no prior topics in your library, so this is your first step into the world of AI. But gradient descent sits at the heart of a whole family of ideas you will meet soon. The most immediate next step is Stochastic Gradient Descent (SGD), which is this same algorithm but running on small random samples of your data rather than the whole dataset — it is faster and, surprisingly, the noise it introduces often helps the model generalise better. You will also meet Momentum, which adds a memory of past steps so the descent can roll through narrow valleys and past small bumps. And you will meet Hyperparameter Tuning, because the learning rate you choose here is a hyperparameter — the most important one, in fact, and getting it right is the difference between a model that learns and one that never does. This concept belongs to the branch of mathematics called optimization — the study of finding the best input to a function, whether that is minimising loss in training or maximising profit in an economy. And the calculus you are about to learn — the derivative — is the tool that tells gradient descent which way to step.