Slice-Based Coverage

This concept is your first tool for making AI evaluation honest. You will build on it with Evaluation Datasets, Gold and Adversarial Sets, Regression Gates, and ultimately the Four-Axis Quality, Safety, Cost and Reliability framework. Without slice-based coverage, you will look at a single accuracy number, think everything is fine, and ship a system that fails for your most important users — the exact shape of a fairness incident. Mastering this unlocks the ability to ask 'where does this break?' instead of 'does this work overall?'

Imagine you are a teacher with a class of 100 students. You give them one test, and the average score is 85 out of 100. That sounds great. But hidden inside that average: 90 students scored between 90 and 95, and 10 students scored 10. The average is 85, yet those 10 students learned nothing. The average 'hid' a group that failed completely. Now imagine you are evaluating an AI system that answers customer questions in three languages — English, Hindi, and Tamil. You test it on 1,000 questions. It scores 90% overall. Great, right? But what if you split the results by language? Maybe it scores 97% in English, 95% in Hindi, and 40% in Tamil. That 40% is a crisis for your Tamil users, but the overall average never shows it. Slice-based coverage means you never look at just the single overall number. You always cut the results into meaningful groups — by language, by customer type, by query category — and check each group separately. The aggregate is a summary, but summaries lie. The slices tell the truth.

Think of a restaurant health inspector. They don't just walk in, taste one dish, and declare the whole restaurant safe. They check the kitchen, the storage, the hand-washing station, the refrigerator temperature — each area separately. One excellent dish doesn't excuse a filthy kitchen. The inspector works by slices, not by an aggregate 'restaurant score'. Now carry this to its limit: the inspector checks many areas because failure in one area has a different consequence than failure in another. If the fridge is broken, that's a different problem than the soup being too salty. Same with an AI system — a failure in the Hindi slice is a different problem than a failure in the English slice, and it needs a different fix. Where the analogy stops working: a restaurant inspector can physically inspect every area in an hour. An AI system might have millions of possible inputs, and you cannot slice every possible group. You must choose the slices that matter — the ones that mirror how your product is actually used. That choice is a judgment call, and it is exactly where the discipline lives. The inspector's job is comprehensive; your job is selective and deliberate.

Slice-based coverage is the practice of evaluating an AI system on meaningful, predefined subgroups of its inputs — such as language, customer type, or query category — and reporting performance for each group separately, instead of relying only on an aggregate metric across everything.

You are starting fresh, so there is nothing to connect to yet. But know this: the library notes you will build reference close neighbours. Evaluation Datasets are the fixed set of inputs you run your slices on. Gold and Adversarial Sets extend this idea — gold sets check correct behaviour, adversarial sets probe failure; both are often analysed by slice. Regression Gates turn your per-slice metrics into a deployment decision: if the Hindi slice drops below 60%, do not ship. And the Four-Axis framework (Quality, Safety, Cost, Reliability) is where slice-based coverage gets applied — you slice each axis separately.