Preference Data Collection

You are building an AI that needs to be reliable, not just fluent. Early training teaches the model to predict words; it does not teach it to follow instructions or avoid harm. To fix this, you need human input to show the model what 'good' looks like. If the human judgments you gather are poor, no amount of computing power will help the model become useful. The bottleneck in creating high-quality AI is not hardware, but the quality of these human comparisons. By the end of this page, you will know how to collect these judgments effectively.

When a model is first trained, it simply predicts the next word. This makes the text flow well, but it does not make the model helpful, honest, or safe. To achieve those qualities, we must show the model what 'good' looks like. We cannot write a precise rule for being helpful because the concept is too vague. Instead, we present the model with many pairs of answers — one correct and one incorrect — allowing it to learn the pattern. This act of gathering those pairs is called preference data collection. It serves as the raw material for alignment. Without it, the model has no definition of 'better.' With it, you can steer the model's behavior in almost any desired direction. The key insight is that these preferences are comparative, not absolute. We do not ask 'is this answer good?' but rather 'is this answer better than that one?' This comparative question is much easier for humans to answer reliably.

Imagine you are a chef teaching an apprentice how to cook. You cannot write a complete recipe for every possible dish, but you can taste two plates side by side and say 'this one is better.' Over many tastings, the apprentice learns your preferences — what you like and dislike. If you consistently prefer the professional chef's dishes over the apprentice's, the apprentice learns to aim for that higher standard. This mirrors how preference collection works: the model is the apprentice, the human annotator is the chef, and each comparison is one data point. A critical detail emerges when the apprentice only tastes their own food; they become more confident but do not improve. Similarly, if a model only compares its own previous outputs, it cannot learn beyond its existing style. This is why collecting data from the model's current generation process is essential — you must evaluate the apprentice's latest dishes regularly, not just the chef's older ones. The analogy has limits, as a chef adjusts in real time while a model learns from thousands of static comparisons, but the core mechanic of learning from pairwise judgments holds true.

Preference data collection is the process of gathering human rankings between different model responses to the same prompt, where humans mark one response as preferred over another, creating the training signal that shapes the model's behavior.

This concept forms the foundation of the post-training stack. You are likely familiar with Supervised Fine-Tuning, which teaches a model to imitate good examples by showing it correct answers. Preference collection builds on this by teaching the model to distinguish better responses from worse ones on its own. The paired data you collect here is consumed by methods such as Direct Preference Optimization and Group Relative Policy Optimization, which adjust the model based on these comparisons. It also feeds into Reward Modeling, a technique that converts these pairwise judgments into numerical scores using the Bradley-Terry model, a statistical method for predicting the probability that one item is preferred over another. The distinction between training on live data generated by the current model versus fixed datasets is crucial here, as is the risk of reward hacking, a situation where the model learns to exploit flaws in the preference data rather than actually improving its helpfulness.