Skill Testing and Evals

You're building skills that an AI will call on by itself. If a skill silently breaks, the whole agent breaks. This concept teaches you how to prove a skill works before it ships — and how to catch the moment a model update quietly destroys something that used to work. It unlocks the rest of LLM evaluation: answer relevancy, deterministic validators, gold and adversarial sets, and finally the discipline of shipping systems that degrade loudly instead of quietly.

Imagine you've written a recipe for the perfect cup of chai. You hand it to a friend. They follow it exactly, and it comes out 'fine but not great.' You taste it: too sweet. Your recipe works when *you* make it, but it fails for your friend. Why? Because you left out 'a pinch of salt' without saying how big a pinch is, or you wrote 'simmer for a while' when you meant precisely four minutes. A skill is exactly this recipe — it packages a repeatable procedure so a different maker (in this case, an AI model) can follow it. But how do you know the recipe works for the maker? You don't — until you test it. Skill testing and evals is the discipline of writing that test. You take a set of inputs that the skill is meant to handle, run the skill on them, and check the outputs match what you expect. You do this before you trust it, and you do it again every time anything changes — the model, the data, the prompt. Because the cost of a broken skill isn't a bad cup of chai; it's an agent that files the wrong tax form, or deletes a database, or tells a customer a confident lie. The intuition is simple: a skill is a claim that 'given X, I will produce Y.' Evals are how you check that claim is actually true.

Think of a quality inspector on a factory line. The inspector doesn't make the product; they check a sample of what comes off the line. They have a checklist: 'Does the lid fit? Does the label face front? Does the button click?' If the sample passes the checklist, the line is trusted to keep running. If it fails, the line stops and someone investigates. That is exactly what skill testing is. Your skill is the factory line; the model that runs the skill is the worker; the checklist is your eval set. You run the skill on a set of pre-chosen inputs, check the outputs against pre-chosen expected answers, and decide whether the skill is good enough to be used. The analogy holds further: a good inspector doesn't just check normal products — they deliberately test the edge cases. Does the lid fit if the jar is slightly deformed? Does the label print if the ink cartridge is low? Translating to skills: does your email-drafting skill work when the input is empty? When it's a thousand-word ramble? When it contains no clear request? Those are your adversarial cases. Where the analogy stops: a factory inspector can check every product. An AI skill is infinite — you cannot test every possible input, so you sample. And unlike a factory, your worker (the model) can change without warning. A new model version might behave differently on the same input. So you re-run the checklist every time the model changes, not just when the skill changes. That's the part of the analogy that's uniquely true for AI: the factory line itself is alive and shifting beneath you.

Skill testing and evals is the practice of running a reusable agent skill against a fixed set of carefully chosen inputs, comparing the outputs to expected behaviour, and using the pass or fail rate to decide whether the skill is safe to use and when it has broken.

This is your first step into LLM evaluation, the parent concept in your library. You have not yet met any of the neighbouring topics — answer relevancy, BLEU, deterministic validators, gold sets, evaluator drift — but this page is the doorway to all of them. The library tells me you've noted: 'Evals are the objective test for a skill just as for a model.' That is the core idea we will make real here. You also noted 'Scope should be tested at the edges, where skills degrade first' — that's the second half of this page. And: 'Untested skills accumulate silent breakage as models change beneath them' — that's the why. If you have not read the 'Claude Skills' material yet, know that this page assumes you know what a skill is (a packaged set of instructions that an agent can invoke). You will also see the word 'deterministic' for the first time — we'll define it fully when it appears.