Repeatability Under Stochasticity

Before you can trust any comparison between two versions of an AI system, you need to know whether the difference is real or just random luck. This concept teaches you how to measure and control that randomness. It unlocks the rest of LLM Evaluation: you cannot meaningfully build benchmarks, regression gates, or evaluator drift detection until you can say how stable your measurements are.

Imagine you are testing two new recipes for biryani. You cook Recipe A once, taste it, and decide it's better than Recipe B. But your aunt cooked it yesterday and found it slightly salty, your cousin found it bland, and the spice blend from the market changed between batches. One tasting is not enough: the dish varies from cook to cook, and that variation can drown out the real difference between recipes. The same happens with AI models. Ask a large language model the same question twice and it may give different answers each time. That is not a bug; it is how these models work. If you compare two versions of a model by asking each one a question once, the difference you see might be the model's true quality difference — or it might be the randomness of the model's responses, or the randomness of your measurement method. Repeatability under stochasticity is the skill of designing your measurements so that the randomness does not fool you. You do this by repeating runs, measuring how much the results scatter, and reporting the scatter alongside the average. Once you know the scatter, you can judge whether the difference between two versions is bigger than the noise.

The darts board is a good picture. Suppose you are comparing two dart players: A and B. You let each throw one dart at a board a meter away. A hits near the bullseye, B hits the outer ring. Clear winner, right? Now suppose you let each throw twenty darts. You find that A's darts are scattered widely — one hits the bullseye, others hit the wall. B's darts, though, all land in a tight cluster just off centre. With one throw each, B lost; with twenty, you see that B is actually the more reliable player — and on average, B's darts are closer to the bullseye than A's scattered hits. The cluster width is the variance: how much a single throw tends to deviate from the average. In model evaluation, each answer is a dart. The model's 'average score' is the target. One run gives you one dart; it tells you almost nothing because it could be an outlier. Ten or twenty runs give you a cloud of darts, and from that cloud you can see where the centre is and how tight the cluster is. The analogy breaks down in one way: model outputs are not physical, independent throws — they are influenced by the prompt, the context, and the model's own parameters — so the scatter is not a fixed property like a player's skill; it changes with the task and the settings. But the central lesson holds: a single measurement is not a measurement until you know how much it wobbles.

Repeatability under stochasticity is the practice of obtaining stable, trustworthy measurements from a system whose outputs vary randomly between runs, by repeating measurements, quantifying the spread of those results, and controlling the sources of variation you can control.

This concept sits at the heart of LLM Evaluation. Before you can compare two model versions, you need this: it gives you the noise floor that tells you whether a 2% difference in BLEU score is meaningful or just random chance. It connects to several neighbouring concepts you have notes on. Deterministic validators are immune to this problem — a schema check gives the same answer every time — so you should rely on them where you can. Reference metrics like BLEU are cheaper and more reproducible than model judges, but they still face stochasticity because the model's output varies. Evaluator drift matters here too: if the judge model updates between two runs, you are not just measuring the model's randomness anymore — you are measuring a different measuring stick. The principle from your notes is that single-run comparisons are noise unless variance is quantified, and that means you cannot build honest regression gates without this concept.