On-Policy vs Off-Policy

This distinction determines how much it costs to train your AI model and what kind of data it needs. When you understand it, you can see why some training methods require a constant stream of new, fresh examples while others are cheaper because they reuse old material. It is the key that unlocks the question of why some approaches demand expensive, real-time interaction with the world, while others simply study static archives.

Imagine you are a chef who wants to learn to cook better. You have two ways to learn. One way is to cook a dish, taste it, and if the food is too salty, you correct it next time. The lesson comes from your own cooking. The second way is to read a recipe book written by a famous chef. You try to copy the book exactly, but you never taste your own attempt; the recipe was created by someone else, in a different kitchen. The lesson comes from an external source.

In AI, we call the model's current strategy for making decisions its 'policy'. When the training data comes from the policy's own current outputs—meaning the model tests itself and learns from those specific results—we call that 'on-policy' learning. This is like the chef tasting their own food. It allows the model to adapt to its own style and immediate environment, but it requires generating new data every time.

When the training data comes from a different source—such as an earlier version of the model, a human expert, or any external archive—we call that 'off-policy' learning. This is like the chef reading a book written by someone else. The lessons come from a fixed record. It can be trained more cheaply because the data is static, but the lessons may not fit the model's current situation or style.

Think of a student driver learning to drive a car. On-policy learning is like taking driving lessons with an instructor in the passenger seat. The student makes a mistake, the instructor tells them immediately ('you are too close to the curb'), and the student learns from their own actions. The data (the driving mistakes and corrections) comes directly from the student's current driving performance.

Off-policy learning is like studying a driving manual written by a professional decades ago. The manual contains general rules ('keep two car lengths behind the vehicle ahead'), but it was written from someone else's experience, not the student's own. The student reads it and tries to follow the rules, but if the student has a specific problem (e.g., a tendency to hug the left side of the road), the manual may not address it, because the book was written for a general audience, not for this particular driver.

The key difference: on-policy uses the student's own driving data to improve; off-policy uses someone else's data. The analogy breaks down slightly because an AI does not have a body or senses—it only knows numbers—but the structural relationship between the source of the feedback and the learner remains the same.

On-policy learning is training a model using data generated by its own current decision-making process, whereas off-policy learning is training it using data from a different source, such as an earlier version of itself or human experts.

This concept sits beside Supervised Fine-Tuning, which is a method where the model learns by copying examples provided by humans rather than generating its own. It also connects to Reinforcement Learning with Human Feedback, a process where the model improves by interacting with a system that judges its responses in real time. On-policy methods typically involve such real-time interaction, while off-policy methods often rely on static datasets.