Bradley-Terry Model

This is the foundation for every modern AI system that learns from preferences. When you ask a chatbot which of two answers is better, or when a recommender system decides what to show you next, or when a model is trained to be helpful rather than just accurate, the math underneath is the Bradley-Terry model. Understanding it unlocks the rest of post-training alignment — reward modeling, RLHF, DPO, GRPO, and the whole machinery that turns human likes and dislikes into a steerable AI. Without this, those concepts are just names; with it, you can trace exactly where the numbers come from and where they go.

Think of a chess rating system. When two players face each other, a stronger player is more likely to win, but not guaranteed — an upset happens, and the bigger the rating gap, the more lopsided the odds. The Bradley-Terry model does the same thing, but for any pair of options, not just chess players. Give each option a single number (its 'strength' or 'score'), and the model says: the probability that option A beats option B depends only on the difference between their scores. If A's score is much higher than B's, A almost always wins; if they're equal, it's a coin flip; if A's score is lower, it usually loses. The magic is that we don't know these scores — we only see the outcomes of pairwise comparisons (A beat B, C lost to D). The model's job is to find scores that make the observed outcomes maximally likely, and it does this by a simple iterative loop: start with guesses, compare each pair's predicted probability to what actually happened, nudge the scores up or down, repeat. After enough iterations, the scores converge to values that best explain the preferences. This is the highway from 'people preferred this answer' to 'here is a number representing how good that answer is' — and that number is what a reward model uses to train an AI.

Imagine you are a cricket match referee who has never seen any team play, but you have a list of match results: India beat Australia, Australia beat England, England beat India (a circular mess). You know nothing about the teams' actual strengths, but you want to assign each team a single 'strength number' so that the results make sense. Start with all strengths equal. Look at the first result: India beat Australia. With equal strengths, your model predicted a 50/50 chance. Since India actually won (a 'surprise' in your model's view), you nudge India's strength up a little and Australia's down a little. Next result: Australia beat England. Now Australia's strength is up, so the model predicts Australia wins more often than 50/50, but not 100% — England still has a chance. When Australia does win, the model is less surprised, so the nudge is smaller. Continue through all results, looping many times. Each nudge moves the scores in the direction that would have made the observed outcome more expected. After enough passes, the strengths settle: India and Australia might be close, England a bit lower. Now you can predict a match between any two teams — and the circular result (England beat India) turns out to be just a lucky upset, not a contradiction. Where the analogy breaks: unlike cricket where a team's strength is roughly consistent, human preferences are inconsistent — people sometimes prefer A over B, B over C, and C over A. The Bradley-Terry model cannot represent that circularity; it will force a single ranking, which may not match every individual preference. Also, the model gives a scalar per item, but real preferences depend on context (a user might prefer short answers for coding, long for essays) — the model ignores that.

The Bradley-Terry model is a statistical method that estimates a single numerical 'strength' for each item from pairwise comparison data, such that the probability one item is preferred over another depends only on the difference in their strengths, found by adjusting the strengths to maximize the likelihood of the observed preferences.

You have not yet studied post-training alignment, so this is your entry point. The concept lives under Post-Training Alignment — everything done after a model has learned to predict the next token to make it follow instructions and behave well. The Bradley-Terry model is the mathematical heart of Reward Modeling, where a model learns to score responses based on human preferences. From there, RLHF uses those scores to fine-tune the language model, and variants like DPO and GRPO skip the explicit reward model but still rely on the Bradley-Terry preference probability. It also connects to Preference Data Collection (the pairwise comparisons are the raw data) and to Reward Hacking (the danger of optimizing the score rather than true quality). Neighbouring topics you will study: Direct Preference Optimization, GRPO, ORPO, Reward Modeling, and Preference Data Collection — all of them build on the ideas here.