ORPO and Preference Optimization

This unlocks the rest of post-training alignment: the family of methods that turn a model that merely completes sentences into one that prefers helpful, harmless, on-task responses. Master this and you can read the papers behind DeepSeek-R1, the alignment stages of Llama and Qwen, and the 'preference tuning' steps in every modern fine-tuning guide. It also gives you the vocabulary to understand Direct Preference Optimization (DPO), GRPO, and the reward-model pipelines that come before them, so you can follow any alignment discussion without getting lost.

A model trained only to predict the next word will happily produce whatever text is most likely — including rude, wrong, or harmful text, because that appears in its training data. Post-training alignment fixes this by teaching the model which responses are *preferred*. The classic way is a multi-stage pipeline: first, fine-tune on demonstrations of good behaviour (SFT), then train a separate 'reward model' to score responses, then use reinforcement learning to push the model toward higher scores. ORPO collapses this into one step. Instead of training a reward model and then running a separate optimization loop, ORPO adjusts the model directly from pairs of responses — one 'chosen' (preferred) and one 'rejected' — during the supervised fine-tuning stage itself. The model learns to assign a higher probability to the chosen response and a lower probability to the rejected one, while also continuing to learn from the chosen response as a demonstration. The result is an aligned model with a single training run, no reward model, and no reinforcement learning loop.

Imagine a new employee learning to write customer support emails. The old way: first, they read a manual of good example emails (that's SFT). Then they take a test where a manager scores each email from 1 to 10 (that's the reward model). Then, to improve their score, they write hundreds of emails and get feedback on each one (that's reinforcement learning). This works but takes months. ORPO is like the manager instead saying: 'Here are two replies to the same customer question — one that a customer loved, one that they hated. Learn to write the loved one and avoid the hated one.' The employee studies the pairs and adjusts their writing style immediately, no separate scoring system. They internalise the preference directly from the examples. Where the analogy breaks down: ORPO doesn't just copy the good email — it *shifts the probability distribution* of all possible replies. It penalises the exact words and style of the rejected one, not just the overall topic. It's a subtle, continuous adjustment, not a discrete 'rewrite'. Also, unlike a human who can generalise from a few examples, ORPO needs many thousands of preference pairs to shift behaviour reliably.

ORPO (Odds-Ratio Preference Optimization) is a single-stage alignment method that updates a language model's weights directly from pairs of preferred and rejected responses, using a loss function that increases the relative probability of the chosen response while decreasing that of the rejected one, all during ordinary supervised fine-tuning.

This concept sits inside the post-training alignment stack. You already have notes on SFT, DPO, GRPO, Bradley-Terry, and reward modeling. ORPO is what happens when you skip the reward model entirely and merge preference learning into the SFT stage — it's a cousin to DPO, but DPO needs a prior SFT step and separate preference-phase, while ORPO does both at once. Your notes on preference data collection and annotation quality are the foundation here: ORPO is only as good as the chosen/rejected pairs it trains on. Your Bradley-Terry notes are relevant because ORPO's loss still encodes a preference probability, but it avoids explicitly fitting a Bradley-Terry reward model. And your notes on reward hacking and on-policy vs off-policy matter because ORPO is off-policy (it trains on fixed preference pairs, not on the model's own current outputs), which makes it computationally cheaper but open to distribution shift, and it is less prone to reward hacking because there is no learned reward to game.