Handling Missing Data

You have a spreadsheet of customer records. Three customers have no listed income. One has no age. Before you can calculate an average income or plot trends, you must decide what to do with these blanks. This decision unlocks the rest of your work—cleaning, transformation, and modeling—because every other step assumes you have handled absence first. Get this wrong, and every number you produce is quietly wrong. In a recent audit, a team ignored missing income values and overestimated their average customer spend by 15%, leading to flawed budget forecasts. In another case, deleting records with missing age skewed the results toward older demographics. This is your first genuine modeling decision: what happens when data is absent?

Think of a spreadsheet where some cells are empty. These aren't just blanks; they are information. The income column might be empty for people who are self-employed because they did not want to share that number. The age column might be empty for people who signed up via an app that never asked for their birthday. The pattern of emptiness reveals how the data was collected, and that pattern matters as much as the values themselves. Before you fill in a gap, ask: why is this gap here? Is it random, like a survey respondent accidentally skipping a question? Or is it systematic, like a sensor that only fails when the temperature is extreme? If income is missing randomly, you might safely fill it with a calculated guess (the average of other incomes). If age is missing because the app never asked, deleting those records entirely might be safe. But if the absence itself is significant—say, people hide their salary when they earn too much—you might treat that absence as a signal rather than a gap to fill.

Imagine you are a chef with a recipe calling for three ingredients: flour, sugar, and eggs. You open the pantry and find you are out of sugar. You have three options. First, you could skip the sugar entirely and proceed—this is dropping the affected records. Second, you could substitute honey that you have on hand—this is replacing the blank with a calculated guess (imputation). Third, you could change the recipe to a sugar-free version altogether, acknowledging that the missingness changes what you are making—this is modeling the missingness. The analogy works because the right choice depends on context. If you are making bread, skipping sugar changes the texture but the bread will still rise. If you are making meringue, skipping sugar means the egg whites will not whip. Where the analogy breaks: in cooking, you can usually taste as you go and adjust. In data, you cannot 'taste' the dataset; you only see the final result after the analysis is done, and by then it is too late to fix a bad choice.

Handling missing data means deciding whether to delete records with gaps, replace empty cells with calculated guesses, or treat the absence of data as a meaningful signal in itself. The choice depends on why the data is missing and what you are trying to measure.

This concept sits inside Data Quality, which explores how missing data patterns carry their own signal. Your notes mention Data Cleaning and Data Transformation; mastering missing data ensures you handle it before scaling or encoding values, avoiding the common error where spreading a problem distorts results. This also connects to Data Validation Constraints, which are the rules that prevent such gaps from appearing in future records.