Alignment Case Study

This is the one place where you see the whole alignment pipeline working together — from raw human preferences to a model that behaves better. You will understand where each piece fits: how preferences become a reward model, how that reward model guides the policy, and how to check honestly whether the result is actually better. It sets you up for deeper topics like RLHF and AI Safety, and for practical skills like building your own preference datasets and evaluation harnesses. If you only study one alignment topic, this is the one that ties everything together.

Think of training a model to be helpful and harmless like training a new employee. You don't just give them a manual and let them loose. You watch them, note what they do right and wrong, and then adjust your feedback to steer them. In AI alignment, the model's 'behaviour' is the text it produces. You collect examples of good and bad responses, and then you train a model to score new responses based on those examples. Then you use that scorer to guide the main model towards better output. The whole process is a loop: generate responses, get them scored, adjust the model, repeat. This case study walks you through that loop end to end.

Imagine you are a movie director with a brilliant but unpredictable lead actor. You want them to perform a scene exactly as you envision. First, you show them examples of the performance you like and dislike (data collection). Then, you hire a critic who has seen all those examples and can instantly judge any new performance as closer to or farther from your vision (reward model). Finally, you rehearse with the actor: they try new ways of delivering lines, the critic tells them what worked, and they adjust their performance bit by bit until the critic is satisfied (policy optimisation). The catch: the critic is not you. They have their own biases, and the actor might find a way to please the critic without actually delivering what you wanted. That is reward hacking. This analogy holds until you realise that the actor is not conscious — it is just a mathematical function adjusting its internal numbers — and the critic is also a mathematical function, trained on a finite set of examples. They don't truly 'understand' the scene; they just optimise patterns. That limit is why alignment is hard and why we need careful evaluation.

An alignment case study is a complete, worked example of taking a language model from its pre-trained state to one that reliably follows a target behaviour (like being helpful or harmless), by collecting human preferences, training a reward model on them, optimising the policy against that reward, and then evaluating the outcome with an eye for both improvements and regressions.

You are learning this from scratch, so we will build every step. The library notes tell me you have heard of terms like 'reward model', 'DPO', and 'RLHF' from other sources, but here we will define them carefully. The case study will draw on the Bradley-Terry model for turning preferences into scores, and it will set up the idea of reward hacking that is central to AI safety. After this, you will be ready to look at RLHF and LLM Evaluation in depth.