Class Imbalance

You are working with data where one outcome is rare—perhaps only 1% of loan applications default, or 0.1% of patients have a specific disease. If you ignore this fact, your predictive model will simply guess the common outcome every time and claim 99.9% accuracy, which sounds impressive but is completely useless for finding the rare cases that actually matter. Understanding class imbalance prevents you from trusting misleading performance scores and gives you the tools to build models that actually detect those critical exceptions.

Imagine you are teaching a child to spot a rare bird. You show them a thousand pictures of pigeons and only ten of the rare bird. After a while, the child learns to say 'not the rare bird' to everything—and they are right 99% of the time. They have learned a rule that is almost always correct, but it is useless. A model does exactly the same thing. When one class vastly outnumbers another, the model can achieve a high accuracy score by always predicting the majority class. The model is not broken; it is doing exactly what it was trained to do—minimise the number of mistakes it makes. But a mistake that misses a rare cancer is not the same as a mistake that mislabels a healthy scan. The cost of errors is asymmetric, but accuracy treats every mistake as equally costly. So the first lesson of class imbalance is that accuracy becomes meaningless long before the model becomes useless. The second lesson is that you cannot just add more data—the imbalance is often a fact of the world, not a data collection error. The third is that you have a toolkit of responses: resample the data so the classes are more balanced, give the model different weights for different classes, and tune the threshold at which the model says 'yes'. Each of these changes what the model optimises, and each has a price. The goal is not to make the model perfectly balanced—it is to make it good at the task that actually matters.

Think of a security guard at an airport. They have to spot a smuggler in a stream of thousands of innocent travellers. If they stop everyone, they catch all smugglers but anger every passenger. If they stop no one, they miss all smugglers but keep the line moving. The guard's job is not to maximise the number of correct decisions in raw terms—it is to balance the cost of a false alarm (a delayed traveller, a lawsuit) against the cost of a missed smuggling attempt (a bomb on a plane). The guard has three tools: they can change how they look at the crowd (resampling—maybe they get more informants, or they ignore some obvious innocents), they can be more cautious by default (class weights—they raise their baseline suspicion), or they can change what they do with a borderline suspicion (threshold tuning—they only arrest when they are very sure, or they arrest on a hunch). The analogy breaks down when you remember that a model does not have judgment or fear—it has a single number that it compares to a threshold. The guard can weigh dozens of signals instantly; a model weighs what you give it, and if you give it only the crowd size and the number of suspicious glances, you have already limited it. Also, resampling is not free—you are discarding data or creating fake data, and that has its own risks. But the guard's core problem—minimising a cost that is not symmetrical—is exactly the model's problem.

Class imbalance occurs when one outcome in your data is much less frequent than another, causing standard prediction methods to ignore the rare outcome because it is easier to always predict the common one.

This concept is central to Feature Engineering, which involves preparing and transforming raw data into a format that a machine learning model can use effectively. It also connects directly to Imbalanced Data, which is the broader category of problems where data distributions are skewed unevenly across any dimension, not just between classes in a prediction task.