Reward Modeling

Reward modeling is the key that unlocks post-training alignment. Without it, you cannot teach a language model to prefer helpful, honest, and harmless responses over unhelpful ones. This concept is the foundation for understanding RLHF (Reinforcement Learning from Human Feedback), reward hacking, and every modern alignment technique like DPO, ORPO, and GRPO. Master this and you'll be able to understand how ChatGPT-style assistants are shaped, how recommendation systems tailor content, and how agents decide which tool to call. It also prepares you for deep dives into AI safety and LLM evaluation, where reward models are the central machinery.

Imagine you are training a puppy. You don't want to give a treat every time it does something, because you can't be there all the time. Instead, you might train a 'judge'—a friend who watches the puppy's behavior and gives a score. That score, a simple number like 0 to 10, tells the puppy (or its trainer) whether the action was good or bad. Reward modeling is exactly this: we train a separate model to 'judge' the quality of a response, producing a single number (the reward) that can be used to steer the main model. The main model is the puppy, and the reward model is the friend with the scorecard. The reward model is not the final goal; it is a stand-in for human judgment, because humans are slow and expensive to ask for every single example. So, we collect a bunch of human preferences (like 'Response A is better than Response B'), and we train a model to predict that preference. Once trained, this model can instantly give a score to any new response, making the training loop fast and scalable.

Think of a movie critic. You, the filmmaker, want to make a movie that audiences love. But you can't screen the movie for every person on Earth. Instead, you hire a critic (the reward model) who has watched thousands of movies and knows what makes a good one. You show the critic your rough cut, and they give you a score out of 10. You use that score to decide which scenes to keep, which to cut, and how to reshoot. The critic is not the audience, but they are a good enough approximation to guide your decisions. This analogy works well, but it breaks down in one important way: the critic can be fooled. If you know the critic loves car chases, you might add a car chase to every scene, even if it ruins the movie. That's reward hacking—optimizing for the critic's score rather than the audience's true enjoyment. The reward model is a compressed representation of what annotators preferred, and like any compression, it loses some information. The model optimizes that compression, not the original intent.

Reward modeling is the process of training a separate model to predict human preference between two or more responses, producing a scalar (single number) reward signal that can be used to guide the optimization of a policy model.

You have not studied any other topics yet, so this is your first step. But I can tell you where this concept sits in the bigger picture. Reward modeling is part of post-training alignment, which is everything done after pretraining to make a model useful and well-behaved. The standard stack is: first supervised fine-tuning (SFT) on demonstrations, then preference optimization using human or model judgements. Reward modeling is the explicit way to do preference optimization: you train a reward model and then use it to fine-tune the policy. Alternative methods like DPO and ORPO skip the explicit reward model and optimize preferences directly, but they are all built on the same idea of turning human preferences into a training signal. This concept also connects to the Bradley-Terry model, which is the mathematical bridge from pairwise comparisons to a scalar score, and to reward hacking, which is what happens when the policy exploits the reward model's blind spots.