Training Dynamics

Training dynamics is the story of what happens inside a model while it learns — how the error (loss) shrinks, how the adjustments (gradients) behave, and how the model's internal representations change step by step. Understanding this will let you: (1) read a training curve and instantly diagnose whether your learning rate is too high or too low, (2) recognize when a model is stuck in a plateau and what to do about it, (3) understand why learning rate schedules are a standard part of every serious training run, and (4) predict when a model might explode or vanish during training. This is the foundation for every later concept: gradient descent, overfitting, regularization, and ultimately how large language models are trained. Without it, you'll be adjusting hyperparameters blindly; with it, you'll know exactly what to change and why.

Imagine you're trying to find the lowest point in a dark, foggy valley. You can't see the whole valley — only the slope right under your feet. You take a step downhill, feel the new slope, and step again. That's training: you repeatedly measure the slope (the gradient), take a small step in the downhill direction, and repeat. The 'loss' is your altitude — how high you are above the valley floor. As you take steps, your altitude should go down, but it doesn't go down smoothly. Sometimes you hit a flat area where the slope is nearly zero — a plateau — and you barely move for a long time. Sometimes you take a step too large and overshoot, landing higher than before — a sharp spike in the loss. Sometimes you stumble into a small dip that isn't the true bottom — a local minimum — and you're stuck there. The 'dynamics' are all about how your altitude and your step sizes change over time. Warmup is when you start with tiny steps to avoid falling off a cliff at the very beginning. A 'schedule' is a plan for how big your steps get as time goes on — big steps early to make progress, small steps later to fine-tune. The shape of the curve you trace as your altitude changes over time is called the loss curve, and it tells you a surprising amount about what's going wrong or right.

Think of training a model like learning to ride a bicycle downhill on a rocky path. The loss is your height above the bottom of the hill — you want to get to the bottom. The gradient is the slope of the ground directly under your front wheel. The learning rate is how hard you push on the pedals each time you feel the slope. If you push too hard (high learning rate), you might speed up, hit a rock, and fly off the path — your loss spikes upward. If you push too gently (low learning rate), you crawl forward and it takes forever to reach the bottom. Early in training, you might be at the top of a steep cliff — if you push hard immediately, you'll crash. So you start with gentle pushes (warmup) and gradually increase your effort as you get your balance. Later, as you near the bottom, you ease off again so you don't overshoot the valley floor — that's a learning rate schedule (like cosine decay or step decay). A plateau is a flat section of the path where the ground is level — you pedal but you don't descend much; you might even think you've reached the bottom, but you haven't — you're just on a shelf. Sharp drops happen when you finally roll off the plateau and the ground suddenly falls away — the loss plummets as you make rapid progress. This analogy breaks down because in real training, you're not moving in a physical landscape — the 'valley' is a high-dimensional space of numbers (the weights). You can't visualize it; you can only feel the slope at your current point. Also, the ground changes as you move — the landscape is not fixed, because the data you're learning from might be sampled randomly. But the core intuition — that you're taking steps downhill and watching your altitude change — is exactly right.

Training dynamics is the study of how a model's loss, gradients, and internal representations change over the course of training, including the effects of learning rate schedules, and how those changes reveal and influence the quality of the final solution.

You haven't learned anything yet — this is your first step. But this concept connects forward to Gradient Descent, which is the algorithm that produces these dynamics. It also connects to Hyperparameter Tuning, because learning rate schedules are a key hyperparameter. In your library, you have notes on Optimization (the parent concept) and neighbors like Convex Optimization and Double Descent — those will make sense later. For now, just know that training dynamics is the observable behavior of gradient descent in action.