In words
What it is, why it matters, and what it is like.
Why am I learning this?
This is the method that actually trains nearly every modern AI model — from a tiny keyword-spotter on a smartwatch to a 70-billion-parameter language model. Without it, none of them could learn from data. Mastering this unlocks: Gradient Descent (the step-by-step process SGD builds on), Momentum (which makes training faster and smoother), and Adam (the optimizer used in practice, which builds directly on these ideas). Understanding SGD is what lets you read any training curve, debug a model that won't learn, and grasp what 'fine-tuning' really does.
The idea, in plain terms
Imagine you are hiking down a mountain in thick fog. You cannot see the whole mountain, only the slope right under your feet. You take a step downhill, feel the ground again, take another step. You will not take the perfect path — you might even step over a small ridge and end up in a different valley — but you will get down, eventually, far faster than if you stopped to survey the entire mountain before every single step.
Training a model is the same. A model has millions of 'dials' (parameters) that decide its behavior. We want to turn those dials so that the model's error (loss) is as small as possible. The slope of the error — the gradient — tells us which direction to turn each dial to reduce the error. But computing the exact slope requires looking at every single example in your dataset, which could be millions of photos, sentences, or transactions. That is expensive — you might only get one step per hour.
Stochastic gradient descent (SGD) says: 'Don't look at everything. Look at a small random handful of examples (a minibatch), guess the slope from just those, and take a step.' The guess is noisy — it could point slightly the wrong way — but you can take a hundred steps in the time it would take to take one perfect step. Over many steps, the noise averages out, and you reach the bottom faster. That is the entire idea in one paragraph.
An analogy
Think of a student trying to master a subject for an exam. There are thousands of practice problems in the book. The 'full gradient' approach would be: solve every single problem perfectly, then figure out which topics you are worst at, then study only that. That is thorough — but it would take months. The SGD approach: grab ten random problems, solve them, notice the topics you got wrong, study those for an evening, then grab ten different random problems, and repeat. Your study plan is noisy — those ten problems might not represent everything — but you are learning every day, not every month.
The student's study plan is the model's parameter adjustments; the ten problems are the minibatch. Over weeks, the student covers all the material many times over, and the noise averages out. The student might even benefit from the randomness: by not studying topics in a fixed order, they avoid 'over-fitting' to the book's chapter sequence and instead learn the underlying logic.
Where the analogy breaks down: a student can reflect on their learning strategy, but SGD has no big-picture awareness. It simply takes a step in the direction of the noisy gradient, every time. Also, the student's goal is a fixed exam; the model's 'mountain' (the loss landscape) can have many valleys, and SGD might end up in a shallow one (a poor solution), while a perfect student would find the global minimum. The noise of SGD can sometimes help escape shallow valleys — a happy accident — but it is not a guarantee.
Definition
Stochastic gradient descent is an optimization method that estimates the gradient of the loss function using a small, randomly selected subset (a minibatch) of the training data, and updates the model's parameters by taking a step in the opposite direction of that estimated gradient, repeating this process many times with different minibatches to gradually reduce the loss.
Where this sits
This is your first step into Optimization, the branch of mathematics concerned with finding the best (minimum or maximum) of a function. Gradient descent is the parent idea — a way to slide downhill. SGD is a specific variant that adds randomness to the gradient to make it faster. You have not met any maths beyond arithmetic yet, but this page will introduce you to two new ideas: a *derivative* (a slope) and an *iterative process* (repeating a step over and over). We will build those from scratch. Soon after, you will meet Momentum (SGD's faster cousin), Adam (which adapts the step size automatically), and Hyperparameter Tuning (which includes choosing the batch size and learning rate, two of the most important dials in training).
