Preference Data Collection

This is the beginning of the alignment stack. Everything after — reward modelling, RLHF, DPO, GRPO, on-policy vs off-policy training, reward hacking — grows from the data you collect here. If the preferences are bad, no algorithm can fix them. By the end of this page you'll know how to gather human comparisons that actually train better models, and you'll understand the single biggest bottleneck in modern AI: not compute or model size, but the quality and quantity of human judgement.

When a model is first trained, it just predicts the next word. That makes it fluent, but not helpful, honest, or safe. To make it those things, we need to tell it what good looks like. But we can't write down a rule for 'being helpful' — it's too fuzzy. So instead, we show the model many pairs of answers — one good, one bad — and let it learn the pattern. This is preference data collection: the act of gathering those pairs. It's the raw material that alignment is built from. Without it, the model has no idea what 'better' means. With it, you can steer the model's behaviour in almost any direction you want. The key insight is that preferences are comparative, not absolute — we don't ask 'is this good?' we ask 'is this better than that?' which is a much easier question for humans to answer reliably.

Imagine you are a chef teaching a new apprentice how to cook. You can't write down a complete recipe for every possible dish, but you can taste two plates side by side and say 'this one is better'. Over many tastings, the apprentice learns your preferences — what you like and don't like. If you taste the apprentice's food first, then a professional chef's food, and say 'the chef's is better', the apprentice learns to aim for the chef's style. This is exactly how preference collection works. The model is the apprentice. The human annotator is the chef. Each comparison is one data point. The critical catch: if the apprentice only ever tastes their own food, they get more confident but don't improve. Similarly, if the model only compares its own outputs, it can't learn anything beyond its own style. That's why on-policy collection matters — you have to taste the apprentice's latest dish regularly, not just the master's old ones. The analogy breaks eventually because a chef can taste and adjust in real time, while a model's preferences are baked in from thousands of static comparisons. But the core idea — learning from pairwise judgements — is exact.

Preference data collection is the process of gathering human comparisons between model outputs for the same prompt, where each comparison marks one output as preferred (chosen) over another (rejected), creating the training signal that shapes a model's behaviour in alignment.

This is the base of the post-training stack. You already know supervised fine-tuning (SFT) teaches the model to imitate good examples. Preference collection goes one step further — it teaches the model to distinguish better from worse on its own. The paired data you collect here is exactly what Direct Preference Optimization (DPO), ORPO, GRPO, and reward modeling all consume. In fact, the Bradley-Terry model, which you'll meet later, is the statistical machinery that turns these raw comparisons into scores. On-policy vs off-policy distinction is central — preference data must come from the policy you're actually training. Reward hacking, the pathology where a model games its reward signal, is only possible if the preference data was badly collected — that's why this concept is the tree from which all RLHF results grow, as your library notes say.