On-Policy vs Off-Policy

This distinction determines how much it costs to train your AI model and what data it needs. When you understand it, you are ready for the entire post-training stack in the learner's library: Supervised Fine-Tuning, Direct Preference Optimization, GRPO, ORPO, and the Reinforcement Learning used in RLHF. It is the key that unlocks the question of why some methods need new data constantly and others are cheap to run.

Imagine you are a chef who wants to learn to cook better. You have two ways to learn. One way is to cook a dish, taste it, and if the food is too salty, you correct it next time. The lesson comes from your own cooking. The second way is to read a recipe book written by a famous chef. You try to copy the book exactly, but you never taste your own attempt; the recipe was created by someone else, in a different kitchen. The lesson comes from an external source. On-policy learning is the first way — the AI learns from its own attempts, like your own tasting. Off-policy learning is the second — the AI learns from data produced by someone or something else, like a recipe book. In the world of AI, the 'policy' is the AI's strategy for making decisions. When the training data comes from the current policy's own outputs, we call that 'on-policy'. When the training data comes from a different, earlier or external source, we call that 'off-policy'. The difference matters because an AI that learns from its own mistakes can adapt to its own style and environment, but that requires it to generate new data each time. An AI that learns from a fixed book of examples can be trained for cheap, but the lessons may not fit its current situation.

Think of a student driver learning to drive a car. On-policy learning is like taking driving lessons with an instructor. The student makes a mistake, the instructor tells them immediately ('you are too close to the curb'), and the student learns from their own actions. The data (the driving mistakes and corrections) comes from the student's current driving. Off-policy learning is like studying a driving manual written by a professional. The manual contains general rules ('keep two car lengths behind the vehicle ahead'), but it was written from someone else's experience, not the student's own. The student reads it and tries to follow the rules, but if the student has a specific problem (e.g., a tendency to hug the left side of the road), the manual may not address it, because the manual was written for a general audience, not for this particular driver. The key difference: on-policy uses the student's own driving data to improve; off-policy uses someone else's data. The analogy breaks down, however, because an AI does not have a body or senses — it only knows numbers. But the idea is the same: the source of the data determines whether the AI is learning from its own current behavior or from a fixed past record.

On-policy learning uses training data produced by the current policy's own outputs, while off-policy learning uses training data from a different, earlier, or external source.

This concept is the foundational distinction behind everything in the post-training stack. Supervised Fine-Tuning (SFT) is off-policy because it uses human demonstrations, not the model's own outputs. Direct Preference Optimization (DPO) and GRPO are on-policy because they sample responses from the current policy and compare them. The Bradley-Terry model is used to turn those comparisons into a reward function, which is then used in on-policy reinforcement learning. Off-policy learning is like learning from a static textbook (cheap, but may be outdated), while on-policy learning is like learning from live practice (expensive, but current). This distinction also drives the operational cost: on-policy methods require generating new data each step, so they are expensive; off-policy methods reuse data, so they are cheap but may drift.