GRPO

GRPO is the technique behind the recent wave of reasoning models — the ones that show their thinking before answering. It is what lets a model learn to reason by comparing its own attempts, without building a separate critic. Understanding GRPO unlocks the rest of post-training alignment: you will see why RLHF needed a value network, why GRPO removed it, and how preference optimisation (DPO, ORPO) relates. It also gives you the vocabulary to read papers and job postings about alignment, and the judgment to know what 'RL-tuned' actually means.

Imagine you are a cricket coach picking players for a team. You don't have a perfect formula for who is good. But you can see the players who performed in the trial match. You put them in groups: say, five bowlers at a time. You watch them bowl. You rank them against each other: this one was better than that one. You don't need an absolute score like 7.3 out of 10 — you just need to know who did better within the group. Then you tell the players: 'you who bowled well, keep doing that; you who bowled poorly, change something.' Over many trials, the team improves.

GRPO is exactly this, applied to a language model. The model sees a question. It samples several different answers (the group). It scores each answer relative to the others — not against an absolute standard, but simply 'is this answer better than the others in this group?' Then it adjusts its weights so that answers ranked higher become more likely, and answers ranked lower become less likely. There is no separate 'critic' model telling it an absolute score; the group itself provides the signal.

Why does this matter? Older methods (like PPO in RLHF) trained a separate critic — a value network — to estimate how good every state was. That doubles memory and compute. GRPO throws the critic away. The group's own relative ranking is the signal. This is much simpler and cheaper, and it turned out to be especially powerful for training models to reason step-by-step, because reasoning tasks often have no single 'perfect' answer — just better and worse paths.

Think of a cooking competition. The judges are not given a scoring rubric out of 10. Instead, five contestants cook the same dish, and the judges simply say: 'this one is better than that one, and that one is the best of the five.' No one says a dish is 'a 7.3' — only which is better than which. That is relative scoring. In traditional RLHF (PPO), the judges would first be trained to give absolute scores ('this dish is a 6.5 out of 10'), and then a separate 'judge model' would be maintained to score every new dish. That is heavy. GRPO says: forget the absolute scores. Just taste the five dishes in each round and rank them.

The analogy breaks down somewhere: in a cooking contest, the judges are human and have consistent taste. In GRPO, the 'judge' is often the same model that is being trained — or a simple rule. For math problems, the judge might just check if the final answer is correct. For code, it might check if the code runs. For open-ended questions, the model itself might be used as a judge (self-rewarding). So the signal can be noisy or even biased by the model's own preferences. But the key insight holds: you don't need an absolute value function; relative comparisons within a group carry enough information to learn.

Group Relative Policy Optimisation (GRPO) is a reinforcement-learning method that improves a model's responses by sampling a group of responses to the same prompt, scoring each response relative to the others in the group (rather than against a learned value function), and updating the model's weights to make better-ranked responses more likely.

You already have notes on post-training alignment, which is the overall stack. GRPO sits in the preference-optimisation stage — after Supervised Fine-Tuning (SFT) and alongside methods like DPO and ORPO. The library notes say: 'the stack is layered: SFT first, then preference optimisation, then RL where warranted.' GRPO is a form of RL where the reward signal comes from relative group comparison.

Your notes mention that DPO and ORPO skip the explicit reward model. GRPO also avoids a learned value network — but it still uses a reward signal per response (e.g., correctness or human preference), which it then normalises within the group. So GRPO is closer to traditional RL than DPO is, but it borrows the relative-comparison spirit of preference methods.

You also have notes on On-Policy vs Off-Policy. GRPO is on-policy: the responses it scores are sampled fresh from the current model at each training step, not from an old version. This is a key property — it means the model always learns from its own current distribution, which reduces the risk of the 'off-policy drift' your notes mention.