LLM as Judge

This concept turns you from someone who can only *use* an LLM into someone who can *verify* it. Once you can build a judge, you can compare two prompts, two models, or two versions of your own app and know which is better — without reading hundreds of outputs yourself. It unlocks the rest of the evaluation stack: building evaluation datasets, catching regressions before users do, and eventually shipping with confidence. From here you can move into Evaluator Drift, Regression Gates, and Four-Axis Evaluation — all of which assume you can score an output automatically.

Imagine you have hired a new assistant. You don't want to read every email they write, but you do want to make sure they aren't making mistakes. So you give a senior colleague a few sample emails and a checklist: 'Does this answer the customer's question? Is it polite? Is it clear?' The senior colleague reads each email, ticks or crosses each item on the checklist, and gives an overall score. That's what an LLM-as-judge does — except the senior colleague is another AI model, and the checklist is called a *rubric*. Instead of you reading a thousand outputs, the judge model reads them all and gives you a score for each. This is useful because modern AI systems generate a huge amount of text — answers, summaries, emails — and checking each one by hand is slow and expensive. The judge makes evaluation *scalable*: you can score a thousand outputs in minutes, not days. But the judge isn't perfect. It has its own opinions, its own biases, and it can change over time. So you have to treat it carefully, just like you would treat a human reviewer who has preferences you need to understand.

Think of a food critic who visits restaurants and writes reviews. You want to know if a new restaurant is good, but you can't visit every restaurant yourself. So you read the critic's review. The critic has a rubric in mind: food quality, service, ambience, value for money. They give each aspect a score and then an overall rating. That's the LLM-as-judge. Now, the critic has preferences — maybe they love spicy food, or they are annoyed by slow service. Those preferences are like the *biases* of the judge model. If the critic changes their taste over time — maybe they become a vegan — their future reviews won't be comparable to their old ones. That's *evaluator drift*. The analogy breaks down because a human critic can explain their reasoning in detail, and you can challenge them. An LLM judge gives you a score and maybe a short explanation, but you can't interrogate it for hidden assumptions. Also, a critic visits one restaurant at a time; an LLM judge can score millions of outputs in parallel — it is far faster but also more opaque.

LLM-as-judge is the practice of using one large language model to evaluate the output of another model (or itself) by comparing the output against a written rubric and producing a score or verdict, instead of relying on human review.

This concept sits inside the broader topic of LLM Evaluation. You already know the idea of Deterministic Validators — programmatic checks that give a clear pass or fail. The judge is different: it handles open-ended quality that a validator can't check. You also know BLEU and Reference Metrics — those compare text by word overlap, but they miss valid paraphrases. The judge is more flexible. Your notes on this topic mention that the judge is a 'versioned dependency' — just like the model you're evaluating, the judge model gets updated, and you need to track that. This connects to Evaluator Drift — the judge's behaviour can change over time, silently breaking your comparisons. And finally, the judge is what powers Regression Gates — automated checks that block a new version of your app if quality drops.