In words
What it is, why it matters, and what it is like.
Why am I learning this?
Understanding evaluator drift is what turns evaluation from a one-time check into a dependable measurement system. Without it, you cannot trust any before-and-after comparison of your AI system. This concept unlocks the ability to build regression gates (knowing when a change actually broke something), monitor models in production (catching silent quality decay), and design trustworthy benchmarks. It is the difference between a score that looks good and a score you can bet a deployment decision on.
The idea, in plain terms
Suppose you are a manager assessing your team's performance. You have a checklist you use every month. But over time, you start marking things differently — you get stricter about punctuality, you care less about presentation, you add a new item about collaboration. Now, when you compare this month's scores to last month's, you cannot tell if the team improved or if your checklist changed. The same thing happens in AI evaluation. The 'judge' that scores your model's outputs — whether it is a human annotator, a rubric, or another AI model — is not a fixed, unchanging ruler. It is a living thing that can change. When the judge changes, the scores change, even if the model itself did not improve or worsen. That is evaluator drift. It silently contaminates every comparison you make, making improvement and regression indistinguishable.
An analogy
Think of a weigh scale in a grocery store. You weigh a bag of apples, get 2 kilograms. Next week you weigh the same bag, get 2.4 kilograms. Did the apples get heavier? Or did the scale drift — its spring loosened, its zero point moved? In a good measurement system, you would check: you put a standard 2-kilogram weight on the scale. If it reads 2.4, the scale is wrong, not the apples. That standard weight is a 'frozen reference'. In AI evaluation, the 'scale' is your judge (model, rubric, human annotators). The 'apples' are the outputs of the system you are evaluating. Evaluator drift is the scale silently changing. This analogy breaks down in one important way: a physical scale drifts slowly and mechanically, but an AI judge can change suddenly (e.g., an update to the judge model that shifts its scoring style) and its changes can be subtle and correlated across many evaluation items. Also, rebuilding the 'standard weight' for AI evaluation is itself a research problem — you cannot buy a certified 2-kilogram weight for 'good answer to a customer query'.
Definition
Evaluator drift is the change over time in the evaluation apparatus itself — the judge model, the rubric, or the human annotators — which has the effect of changing scores without any change in the system being evaluated, thereby invalidating any comparison of scores across time.
Where this sits
This concept builds directly on LLM as Judge, where a model scores another model's output — that judge model is the most common source of drift. It also connects to Evaluation Datasets, because a stable dataset is part of the frozen reference you need to detect drift. Your library notes this concept belongs to the parent concept LLM Evaluation, and that a key idea there is that the judge is a 'versioned dependency'. Neighbouring topics that relate: Deterministic Validators are immune to drift because they are programmatic and always produce the same answer, making them useful as a stable anchor; BLEU and Reference Metrics drift less (the formula is fixed) but are blind to paraphrase, so they cannot serve as a complete reference. Benchmarking suffers from a related problem: public benchmarks get contaminated by training data, which is another form of the evaluation reference changing.
