Gold and Adversarial Sets

This unlocks the study of evaluation in LLM applications. One you can tell a correct answer from an incorrect one, you can read about answer relevancy and correctness, benchmarks, deterministic validators, and the eval gap. Understanding gold and adversarial sets is the first step to building an evaluation suite that actually protects your product.

Imagine you are a teacher. To know if a student has learned, you don't just ask one question. You ask a set of questions that you know the correct answers to. That set is a 'gold set' because it captures the expected behaviour. But a good teacher also tries to trick the student. You ask questions that look easy but require careful thinking, or that test whether the student really understands the concept rather than just memorised the facts. That second set, the tricky ones, is the 'adversarial set'. Both are needed: the gold set shows that the student can handle normal questions, and the adversarial set shows where the student is weak. In AI, a 'gold set' is a list of inputs with the exact expected answers, and an 'adversarial set' is a list of inputs designed to break the model – to catch the model making a mistake when it shouldn't. If you only test with easy questions, you don't know if the model is truly capable. If you only test with tricky ones, you don't know if it can handle everyday use. You need both.

Think of a cricket team. You want to pick a team for the World Cup. You don't just watch them play against a weak local club – that tells you nothing. You watch them play against top international teams, and you also watch them play against teams that use unconventional tactics – like a 'mystery spinner' who bowls a delivery that looks like a normal ball but dips in the air. The matches against strong teams are like your gold set: they measure whether the team can perform well in standard high-pressure conditions. The matches against the mystery spinner are your adversarial set: they probe for a specific failure – 'can they read the ball out of the hand?' If they fail that, you know they will struggle against a certain type of bowler. Similarly, a gold set for an AI model asks it to do typical tasks with correct answers, and an adversarial set asks it to do tasks that are designed to expose a weakness – for example, asking a chatbot for a legal advice when it is not confident, or asking a summarisation model to summarise a text that contains contradictory information. The adversarial set doesn't have to be realistic – it is meant to be a stress test. But it must be based on real failure modes you have observed or anticipate, not just random weird questions.

A gold set is a curated collection of inputs with known correct outputs, used to measure whether a system performs its intended function; an adversarial set is a curated collection of inputs deliberately designed to trigger failures, used to probe the boundaries of a system's capability.

This concept is the foundation for everything else in evaluation. You have not yet studied it, but your library notes mention it as part of LLM Evaluation. Once you have built gold and adversarial sets, you can use them to measure answer relevancy (is the answer addressing the question?) and correctness (is it true?). You can use them to compute BLEU scores or other reference-based metrics. You can use them to run deterministic validators (schema checks, forbidden content) on the outputs. You can use them to set up regression gates – if a new version of your model performs worse on the gold set, you don't deploy it. So this concept is not just about collecting data; it is the bedrock of the whole evaluation discipline.