BLEU and Reference Metrics

You will use BLEU and similar metrics as your first tool for judging whether a machine-generated sentence is any good. Before you can build a system that translates, summarizes, or answers questions, you need a way to say 'this output is close to the right answer' or 'this output is nonsense' — and you need it to be automatic, cheap, and reproducible. BLEU is one of the oldest and simplest such tools. Learning it here gives you the foundation for everything that follows: you will later meet more advanced metrics, learn why BLEU is often not enough, and move on to human judgment and AI-as-judge. But BLEU is where you start because it is simple enough to understand completely from scratch, and it teaches you the core idea that most evaluation metrics share: compare what the machine produced against a known-good reference, and measure how much they overlap. Once you know that pattern, you will recognize it everywhere, and you will know why it breaks when you push it too far.

Imagine you are a teacher grading a student's essay. The student was told to write a short summary of a paragraph you gave them. You have your own 'perfect' summary in mind — the answer key. How do you decide the student's summary is good? You compare it to your answer key. If the student used the same important words, in roughly the same order, you'd say it's a good summary. If they used completely different words, you'd say they missed the point. BLEU automates exactly this. It looks at the generated text and the reference text, and counts how many small chunks of words (called n-grams) appear in both. The more overlap, the higher the score. It's like a matching game: you take every pair of adjacent words in the generated text (a 2-gram, a bigram), and you check if that same pair exists in the reference. You do the same for single words (1-grams), for triplets (3-grams), and so on. The final BLEU score is a number between 0 and 1 (or 0 and 100), where 1 (or 100) means the generated text is identical to the reference in terms of n-gram overlap. If there's no overlap at all, the score is 0. That's the whole idea: measure overlap. It's crude, but it's a start, and it's something you can compute in a fraction of a second, on any text, with no human involvement.

Think of BLEU like a teacher checking a student's answer against an answer key, but a very literal, mechanical teacher who only checks that the same words appear in the same order — not the meaning. You have two pieces of paper: the student's essay and the teacher's model answer. The teacher goes through the student's essay and for every pair of adjacent words, she draws a circle around that pair and checks whether that exact same pair appears anywhere in the model answer. If yes, she puts a tick. If not, a cross. She does the same for single words, and for triplets. At the end, she counts up the ticks and crosses. The score is the proportion of circles that got a tick. Now, this teacher has no sense of synonyms. If the student writes 'the cat sat' and the model answer says 'the cat slept', the teacher sees the pair 'the cat' — tick. But 'cat sat' doesn't appear, so that pair gets a cross. Also, the teacher is easily fooled: if the student writes 'the the the', the teacher sees 'the' appearing three times, and if 'the' is in the model answer (it almost certainly is), those single-word pairs would get ticks, even though the sentence is gibberish. That's the flaw of BLEU: it counts overlap, not sense. Where does the analogy break down? A real teacher would understand that 'feline' is the same as 'cat', that word order changes meaning, and that a short but perfect answer is better than a long padded one. BLEU doesn't understand any of that. It just counts. Also, the teacher would consider the whole answer, while BLEU has a 'brevity penalty' that punishes answers that are too short, but it doesn't punish repetition. So the analogy holds for the basic mechanism, but BLEU is a glorified word-counter, not a judge of understanding.

BLEU (Bilingual Evaluation Understudy) is an automatic metric that scores how well a machine-generated text matches one or more reference texts, by counting the overlap of n-grams (contiguous sequences of n words) between them, with a penalty for outputs that are shorter than the references.

This concept sits at the very beginning of your learning about evaluation. You have no prior mastery of any other topic, so this is the first brick. In your library's notes, BLEU is part of LLM Evaluation, alongside tools like 'Answer Relevancy and Correctness' and 'Deterministic Validators'. You will later learn that BLEU is a weak judge of quality — it can't tell if a paraphrase is good — but it is a fast and reproducible one, which is why it's still used in production as a quick regression check. You'll also learn that modern AI evaluation often uses 'LLM as judge' models, but those need to be calibrated against a gold standard, and BLEU is one of the classic gold-standard baselines. So think of BLEU as your first stepping stone: it's simple, it works, and it teaches you the core evaluation loop (define reference, compare, score) that everything else builds on.