In words
What it is, why it matters, and what it is like.
Why am I learning this?
You are building automated assistants that perform tasks on your behalf. If one of these helpers makes a quiet mistake, the entire workflow fails without warning. This concept teaches you how to verify that a helper actually does what it claims before you let it go, and how to catch the moment a software update silently ruins a task that used to work perfectly. It is the foundation for checking if an answer fits the request, whether checks run consistently, and how to build systems that fail loudly so you can fix them quickly. For example, knowing that a skill has passed 95 out of 100 test cases gives you concrete confidence; failing 8 out of 10 tells you exactly where it breaks.
The idea, in plain terms
Imagine you have written down a recipe for the perfect cup of chai. You hand it to a friend. They follow every step exactly as written, but the result is 'fine but not great.' You taste it and realize it is too sweet. Your recipe works when *you* make it because you intuitively know how big a 'pinch' of salt is or what 'simmering for a while
An analogy
Think of a quality inspector on a factory line. The inspector does not make the product; they check a sample of what comes off the line. They have a checklist: 'Does the lid fit? Does the label face front? Does the button click?' If the sample passes the checklist, the line is trusted to keep running. If it fails, the line stops and someone investigates. That is exactly what skill testing is. Your skill is the factory line; the model that runs the skill is the worker; the checklist is your eval set. You run the skill on a set of pre-chosen inputs, check the outputs against pre-chosen expected answers, and decide whether the skill is good enough to be used. The analogy holds further: a good inspector does not just check normal products — they deliberately test edge cases where things might go wrong, such as a lid that fits tightly because the jar is slightly deformed or a label that smudges if the ink cartridge is low. In skills testing, these are inputs designed to break the logic, like an empty email request or a thousand-word ramble with no clear instruction. Where the analogy stops: a factory inspector can check every product. An AI skill is infinite — you cannot test every possible input, so you sample. And unlike a factory, your worker (the model) can change without warning. A new model version might behave differently on the same input. So you re-run the checklist every time the model changes, not just when the skill changes. That is the part of the analogy that is uniquely true for AI: the factory line itself is alive and shifting beneath you.
Definition
Skill testing and evals is the practice of running a reusable agent capability against a fixed set of carefully chosen inputs, comparing the results to expected behavior, and using the pass or fail rate to decide whether the capability is safe to use and when it has broken.
Where this sits
This is your first step into LLM evaluation, the parent concept in your library that covers how to judge if a model is doing what you want. You have not yet met its neighboring topics — answer relevancy (how well the output matches the user's specific question), deterministic validators (rules that check for obvious errors like formatting or missing data without ambiguity), and gold sets (a curated list of perfect answers used as the standard for comparison). The library tells me you've noted: 'Evals are the objective test for a skill just as for a model.' That is the core idea we will make real here. You also noted 'Scope should be tested at the edges, where skills degrade first' — that's the second half of this page. And: 'Untested skills accumulate silent breakage as models change beneath them' — that's the why.
