Null and Duplicate Handling

This is the first gate you must pass before any of the interesting AI work can happen. Every dataset you will ever touch — whether it comes from a company's internal records, a government database, or a scraping script — arrives with missing cells and repeated rows. If you do not handle these two defects, every number you compute afterwards is distorted: averages are wrong, models learn from data they should never have seen, and evaluations look far better than real-world performance. Learning this concept unlocks the rest of the Data Quality track: Handling Missing Data (where you decide whether to drop, fill in, or model the absence itself), Outlier Handling (identifying extreme values), and Constraint Generation (automatically discovering rules to keep future data clean). It also builds the foundation for building any model — because training a model on a dataset with duplicates is like teaching a student using the same book twice while skipping the others.

Think of a dataset as a table with rows and columns, like an Excel sheet. Each row is one record — one customer, one transaction, one patient visit. Each column is one attribute — age, income, date-of-purchase. Now imagine you've been given a spreadsheet of sales transactions from a small shop. You open it and notice two things: some cells are empty (maybe the customer didn't provide their phone number), and some rows look identical (maybe the same sale was recorded twice by mistake). Both defects are not just cosmetic — they corrupt what you can learn from the data. An empty cell means you have less information about that customer; if you want to compute the average purchase amount, an empty amount forces you to decide whether to ignore that row, fill in a guess, or treat it as zero. A duplicate row means that customer is counted twice — inflating their apparent buying behaviour, and if you split your data into training and testing sets carelessly, the same customer can appear in both, letting the model 'cheat' by memorising rather than learning.

The key intuition is that both defects are structural: they are not about the data's content being wrong, but about its shape being wrong. Handling them is the first step in any data-cleaning process, and it happens before any analysis or model-building. The skill is not just about running a command to delete rows — it's about understanding what each empty cell and each repeated row means in your specific context, because the right action differs: sometimes you drop, sometimes you fill, sometimes you keep the duplicate on purpose.

Imagine you are a librarian given a box of index cards, each card describing a book in the library: title, author, publication year, shelf number, and a checkout count. But the previous librarian was sloppy. You find two sets of problems. First, some cards have a blank space for the shelf number — maybe the book is lost, maybe the librarian forgot to write it, maybe the card itself is from a time when the book wasn't in the library yet. You cannot put a book away without a shelf number, so you must decide: do you set the card aside, do you write 'UNKNOWN' on it and hope someone finds the book later, or do you guess a shelf number based on similar books? Each choice has consequences: setting it aside loses access to that book's checkout history; guessing might put it on the wrong shelf, making it unfindable. Second, you find that some cards are exact duplicates — the same book described twice, perhaps written on two different days. If you count the duplicate, the library's total book count is inflated, and if you use the checkout counts to decide which books are popular, the duplicate doubles the apparent popularity of that title. So you must decide: do you merge the two cards, keep one and discard the other, or — if you know the duplicate is real, because the book somehow has two physical copies — keep both? The analogy breaks down when you realise that in a library, you can physically inspect the cards and the books to resolve ambiguity. In a dataset, you often have no way to know why a cell is empty or why a row is duplicated — and the decision you make changes the outcome of every analysis downstream. That is why null and duplicate handling is not a mechanical task; it is a judgement call, informed by your understanding of where the data came from.

Null and duplicate handling is the set of decisions and actions taken to manage absent values (nulls) and repeated records (duplicates) in a dataset — determining what each absence and each repetition means, and then dropping, filling, merging, or keeping rows and cells accordingly, so that downstream analysis and modelling rest on data free of these two structural defects.

This concept sits at the very start of the Data Quality track. Before you can do any exploratory data analysis, you must know which cells are empty and which rows are repeated — otherwise your first plot or first average is already wrong. The actions you choose here feed directly into Handling Missing Data (which you'll study next), because null handling is usually the first half of that topic: you decide whether to drop, impute, or model the missingness. Duplicate handling connects to Constraint Generation, because after you've fixed a dataset once, you can generate rules ('no duplicate customer IDs', 'no empty price cells') to prevent the same defects from recurring in future data. And it connects to Data Validation Constraints, because those constraints are the automated enforcement of the decisions you make here. The learner's library notes also mention that duplicates leak between train and test splits — a key reason to deduplicate before splitting, which you'll develop further when you study model evaluation.