Imbalanced Data

This is the gateway to every real-world ML project. Before you can build a spam filter, a fraud detector, or a medical triage system, you must handle the fact that the rare event is the one that matters. Master this concept and you unlock the ability to evaluate models honestly, fix skewed training data, and avoid the trap where a model looks great but does nothing useful. It directly prepares you for 'Class Imbalance', 'Feature Selection', and eventually 'MLOps' — no serious model is built without confronting this problem first.

Imagine you are a security guard at a stadium entrance. You see thousands of fans a day, all with tickets. But every once in a while, someone tries to sneak in with a fake ticket. If you focus only on the 'normal' fans, you become great at letting them through, but you completely miss the fakes. An AI model does the same thing. It learns from the data you give it. If 99% of your data is 'normal' and only 1% is 'fake', the model quickly figures out it can score a 99% accuracy just by always predicting 'normal'. It becomes lazy. It never learns the subtle signs of a fake ticket. This is imbalanced data. It happens everywhere: fraud is rare, diseases are rare, equipment failures are rare. The problem isn't the rarity itself; it's that the model's goal — to minimise error — is misaligned with your goal, which is to catch the rare event. The model minimises error by ignoring the minority, but you don't care about the 99% it got right; you care about the 1% it missed.

Imagine you are a talent scout for a cricket team, and you have to decide which players are good enough to move up. You watch hundreds of local matches. But only a handful of players are truly exceptional — say 5 out of 500. If you just watch the matches and take notes, your brain naturally remembers the good players (the majority, who are average) and the exceptional ones (the minority) stand out. But an AI model training on your match notes would be like a scout who only records whether a player scored a century. If only 1% of players score a century, the model learns that 'century' almost never happens. So it predicts that no one will score a century. It's technically right 99% of the time, but it's useless to you. You want the model to find the 1% who score centuries, even if that means it sometimes flags a few average players as exceptional. That is the core tension: you accept more false alarms to catch the rare true positive. This analogy starts to break down, though, because a human scout uses intuition and context, whereas a model has no intuition — it only has numbers. The model cannot 'want' to find the rare player; it only does what the loss function tells it to do. That's why you, the engineer, have to change the loss function or the data.

Imbalanced data is the situation where the classes (or categories) in a training dataset are not represented equally, with one class appearing much more frequently than others, which biases the model to favour the majority class unless corrected.

This concept sits at the heart of Feature Engineering, because the choice of features and the balance of the data interact strongly. Your notes on 'Class Imbalance' are the direct follow-up: it's the concrete problem of one class outnumbering others, and this page is the foundation for that. This concept also connects to 'Data Quality' — an imbalanced dataset is often a sign of poor data collection, not just a mathematical quirk. And it connects to 'Feature Selection' — if you have imbalanced data, some features may be unhelpful for the minority class, so you need to choose features that help separate the rare cases, not just the majority.