Handling Missing Data

Handling missing data is the first real decision you make in almost every data project. Before you can train a model, plot a trend, or compute an average, you must decide what to do with the gaps. This unlocks the rest of the Data Quality path — Data Cleaning, Data Transformation, and eventually Model Monitoring — because every downstream method assumes you've already dealt with absence. Get this wrong and every number you produce afterwards is quietly wrong; get it right and you've made your first genuine modelling decision.

Think of a spreadsheet of customer records, where some cells are empty. The empty cells aren't just blanks — they're information. Maybe the income column is empty for people who are self-employed, because they didn't want to share. Maybe the age column is empty for people who signed up via a certain app that never asked. The pattern of emptiness often reveals how the data was collected, and that pattern matters as much as the values themselves. Before you fill in a gap, you need to ask: why is this gap here? Is it random, like a survey respondent accidentally skipping a question? Or is it systematic, like a sensor that only fails when the temperature is extreme? The answer determines whether you can safely fill in the missing value with a guess, or whether you should drop the record entirely, or whether you need to treat the absence itself as a feature.

Imagine you're a chef with a recipe that calls for three ingredients: flour, sugar, and eggs. You open the pantry and find you're out of sugar. You have three options. First, you could skip the sugar entirely and proceed — that's dropping the missing ingredient. Second, you could substitute honey that you have on hand — that's imputation, replacing the missing value with a stand-in. Third, you could change the recipe to a sugar-free version altogether, acknowledging that the missingness changes what you're making — that's modelling the missingness. The analogy works well because it shows that the right choice depends on context. If you're making bread, skipping sugar changes the texture but the bread will still rise. If you're making meringue, skipping sugar means the egg whites won't whip. Similarly, dropping a missing value is fine when the record is unimportant, but dangerous when the missingness is tied to the outcome you care about. Where the analogy breaks: in cooking, you can usually taste as you go and adjust. In data, you can't 'taste' the dataset — you only see the final result after the model is trained, and by then it's too late to fix a bad imputation choice.

Handling missing data is the process of deciding what to do about absent values — dropping the affected records, imputing replacement values, or modelling the missingness itself as a signal — based on why the data is missing and what analysis you're about to run.

This concept sits inside Data Quality, which you've already begun exploring. Your notes say that 'missing data is information — the pattern of absence often carries signal.' That's exactly what this page builds on. You also have notes on Data Cleaning and Data Transformation, both of which you'll use constantly after mastering missing data. In particular, Data Transformation's warning that 'order of operations changes results, particularly around missing values' is a direct consequence of what you'll learn here — always handle missing values before you scale or encode, or you'll spread the problem. This concept also connects to Data Validation Constraints: constraints are how you prevent missing values from sneaking into future data.