Answer Relevancy and Correctness

This concept is the foundation of LLM Evaluation — the discipline of measuring whether a generative or agentic system works. Every evaluation you will later build — using reference metrics like BLEU, running benchmarks, or deploying LLM-as-judge — depends on distinguishing two kinds of failure: an answer that misses the question (relevancy) versus an answer that gets the facts wrong (correctness). Mastering this distinction unlocks the ability to design evaluation datasets, build validators, and interpret model scores meaningfully. Without it, you cannot tell whether a system that 'responds well' is actually answering what was asked or merely producing fluent-sounding but empty or false text. This is the first step toward the broader skill of evaluating AI systems across the four axes of quality, safety, cost, and reliability.

Imagine you ask a store assistant: 'Do you have a blue cotton kurta in size M?' Now consider three possible replies:

1. 'We have kurtas in the back, but I think they're all polyester, and we only go up to size L.'
2. 'Yes, we have blue cotton in size M — right over there on the third shelf.'
3. 'The weather is lovely today, and the parking lot is almost full.'

Reply 1 addresses the question — it talks about kurtas, sizes, and fabric — but it does not give a clear yes or no to your specific request. It is somewhat relevant but not fully correct in answering.

Reply 2 is both relevant (it directly addresses the request) and correct (it confirms availability and location — but only if you trust the assistant's knowledge).

Reply 3 is neither relevant nor correct — it ignores the question entirely.

Now two separate failures exist: irrelevancy (the answer does not address the question) and incorrectness (the answer makes a false claim). An answer can be relevant but false: 'Yes, we have that kurta in red' — relevant because it speaks to the request, but false if the red kurta is out of stock. An answer can be true but irrelevant: 'The store closes at 9 PM' is true, but it does not answer the question. This distinction is the core of evaluating LLM-generated answers: you need to check the answer against the question (relevancy) and against facts or evidence (correctness) independently.

Think of a customer service representative on a phone line. You call and ask: 'What is the refund policy for a damaged item?'

The representative has two distinct jobs: to understand your question (relevancy) and to give you accurate information (correctness). These are separate skills, and they can fail independently.

A representative who says 'Our return window is 30 days' is being relevant — they are talking about returns, which is close to your question — but if the actual policy is 14 days, they are incorrect. A representative who says 'We have a loyalty program with points' is being irrelevant — they heard 'policy' and went off on a tangent — even if every word they say about the loyalty program is factually true.

Now push this analogy to its limit: the representative is not a human but a machine that has read millions of customer-service transcripts. It has learned patterns of what a 'good answer' looks like — fluent, polite, on-topic-sounding — but it does not have a fixed database of policies. Its 'knowledge' is a statistical guess about what words usually follow. This is where the analogy breaks down and becomes crucial: the machine can produce an answer that sounds perfectly relevant and confident but is entirely fabricated. A human representative, when unsure, can say 'Let me check'; a language model cannot easily say 'I don't know' unless it has been specifically trained to. This is why relevancy and correctness are evaluated separately: a fluent-sounding answer that addresses the question can still be completely wrong in its facts, and vice versa. The evaluation must catch both failures, and it needs different methods for each.

Answer relevancy measures whether a generated response addresses the question that was asked; answer correctness measures whether the claims in the response are true, grounded in evidence or a reference. They are two independent axes of quality, and an evaluation system must score them separately.

This concept is the entry point to the broader discipline of LLM Evaluation. It connects to neighbouring topics in your library: BLEU and Reference Metrics compares a generated answer to a gold-standard reference by word overlap — but it cannot judge whether the answer is relevant to the question, only whether it resembles a fixed reference. Deterministic Validators give unambiguous pass/fail on format and structure, which is the correct layer for catching irrelevancy of a mechanical kind (like answering in the wrong language) but never for judging truth. LLM as Judge uses a model to score an output against a rubric — and here the distinction between relevancy and correctness becomes the heart of the rubric design; if you conflate the two, the judge produces a single meaningless score. Gold and Adversarial Sets require you to label expected behaviour — and labelling relevancy vs correctness separately is how you build a dataset that can expose both failure modes. The book *Transformers and Large Language Models* notes that RAG quality is decomposed into measurable axes — context precision, context recall, answer relevancy, answer correctness — rather than a single score, and that decomposition is exactly what this concept enables.