In words
What it is, why it matters, and what it is like.
Why am I learning this?
Understanding the difference between offline and online evaluation is the foundation for everything that follows in LLM Evaluation. It determines how you know whether a model change is an improvement or a regression before real users see it. This unlocks: Regression Gates (converting metrics into deployment decisions), Evaluator Drift (how the evaluation itself can change), and Model Monitoring (detecting problems in live traffic). Without this, you will ship changes that break your product without knowing.
The idea, in plain terms
Offline evaluation is like a dress rehearsal: you run the model on a fixed set of scripted scenes before the real show. It is fast, safe, and repeatable, but the audience isn't there yet, so the scenes may not match what the audience actually says. Online evaluation is the opening night: you watch real people react to the model in real time. It is the truth, but any mistake is seen by everyone, and fixing it after the fact is expensive. In practice, you rehearse first — you gate what gets to opening night on how the rehearsal went — but you also watch the opening night, because the rehearsal can never cover every surprise. A mature system runs both, in a sequence: offline gates whether something is even considered, online tells you the final truth about whether it actually worked.
An analogy
Think of a chef testing a new recipe before putting it on the menu. Offline is the chef cooking the dish in the kitchen with their own taste-testers, checking it against a fixed list of criteria: is it too salty, does it look right, does it match the plating guide? This is fast, cheap, and safe — the dish never reaches a paying customer. But the taste-testers are not the real diners; they may not represent what the public actually likes. Online is serving the dish to actual customers for a limited time and watching what they order, what they send back, and what they criticise in reviews. This gives the true answer, but a bad dish can harm the restaurant's reputation. The kitchen's job is to use offline tests to filter out obviously bad recipes, then test the promising ones online on a small scale before a full rollout. The analogy breaks down here: in a restaurant, the offline and online tests happen days or weeks apart, and the kitchen can't easily revert a dish once it's on the menu. In software, you can switch back instantly and at low cost, and you can run offline and online simultaneously — so the sequence is even more powerful, but the principle remains: each has a role, and neither alone is enough.
Definition
Offline evaluation measures a model's performance against a fixed, pre-recorded dataset before it is released, while online evaluation measures how the model performs on live, real-world traffic after it is deployed.
Where this sits
This concept builds on nothing yet — it is your starting point in the LLM Evaluation track. It sits alongside Evaluation Datasets: you cannot run offline evaluation without a fixed dataset, and that dataset is what makes two model versions comparable at all. It connects forwards to MLOps and Model Monitoring, where online evaluation feeds continuous checks on live systems. It also relates to Training-Time vs Inference-Time Evaluation, which you will study later: offline evaluation is a kind of inference-time evaluation on a fixed set, while online is entirely inference-time on live traffic.
