Benchmarking

Benchmarking is the measuring stick of AI. When someone says ‘this model is better than that one’, they are almost always pointing at a benchmark score. If you want to know which model to choose for your task, whether a new version is actually an improvement, or how to talk about AI performance with any credibility, you need to know what those numbers mean — and what they don't. This page gives you that foundation, so that when you later work with evaluation, safety benchmarks, or model monitoring, you already know the ground rules of comparison.

Suppose you want to buy a new phone. You have two models in mind, and you want to know which one takes better photos. You could just try them both yourself, but that takes time and might be biased by your own preferences. Instead, you look at a photo review site that takes a standard set of pictures — a sunlit landscape, a dim indoor shot, a portrait — with each phone, and then compares the results. That standard set of pictures is a benchmark. It gives both phones the same task, under the same conditions, so you can compare them fairly. In AI, a benchmark is exactly that: a fixed, standardised set of tasks that many different models are asked to perform, so that their results can be compared. The score a model gets on that benchmark is its report card — but report cards only measure what the test actually covers. If the test is all about maths, you don't learn which student writes better essays. In the same way, a benchmark that tests one kind of task tells you nothing about performance on a different kind of task.

Think of benchmarks as international sports competitions, like the Olympics. Every athlete from every country competes in the same events, under the same rules, and their performances are measured with the same stopwatches and tape measures. This gives a common frame of reference: a gold medal in the 100-metre sprint means the same thing whether the winner is from India or Brazil. That is exactly what benchmarking does for AI models. The model is the athlete, the test tasks are the events, and the score is the medal. But the analogy has an important limit: athletes train specifically for their events. A sprinter who wins gold in the 100-metre does not automatically win the marathon. In AI, a model that scores brilliantly on a benchmark may perform poorly on a real-world task that is slightly different in style or context. Moreover, athletes and coaches study past Olympics to train for the next one — and if the same events are reused year after year, competitors start to train specifically to game the test, rather than to improve their true ability. That is contamination: when a model has seen the benchmark questions during its training, its score stops being an honest measure and becomes a memory test. So the Olympic analogy holds until the point where the test itself becomes part of the training — and then it breaks down, because the benchmark no longer measures what it claims to measure.

A benchmark is a fixed, standardised suite of tasks, each with a known or expected answer, used to compare the performance of different AI models, but its score only reflects performance on that specific set, not on your real-world problem.

This page is your first step into the wider topic of LLM Evaluation — the art of measuring whether a generative or agentic AI system actually works. The learner's library records that benchmarking sits under that parent concept, alongside neighbouring topics like BLEU and Reference Metrics (which are automatic scoring methods used inside benchmarks), Evaluation Datasets (the curated collections of examples that benchmarks are built from), and LLM Safety Benchmarks (specialised suites for harmful behaviour). Once you understand benchmarking, you'll be ready to dive into those. A benchmark is essentially a pre-made evaluation dataset with a scoring rule attached. Everything you learn here — contamination, the gap between benchmark and real task, and the importance of your own evaluation — will apply directly when you build your own sets.