Entity Resolution

You will learn what it means for two rows of data to be about the same person, the same company, or the same product — even when the rows come from different systems and share no common ID. This is the skill behind fixing duplicate customers in a CRM, connecting a bank's records to a third-party credit bureau, and merging data after a merger. It unlocks the next topics in your library: Data Pipelines, Big Data Processing, and the Data Fabric and Mesh note you already have, because entity resolution is the part of building them that people most often underestimate.

Think of the last time you tried to find a friend on a social network. You typed a name, got several profiles, and had to decide which one was actually them. Maybe one had the right name but a photo from a different city. Another had the same job title but no photo. A third matched the name and the city, but had a different last name because they had recently married. You combined clues — name, location, appearance — and made a judgment call. Entity resolution is automating that judgment call for millions of records at once. It is the process of deciding when two rows of data refer to the same real-world thing, like a person, a company, or a product, when neither row carries a unique identifier you can just match on. In a perfect world, every system would use one standard ID, like an Aadhaar number for people or a GSTIN for businesses. In reality, the same company might be 'Reliance Industries' in one database, 'RIL' in another, and 'Reliance Industries Ltd.' in a third. The same customer might be 'A. Sharma' in one system and 'Anita Sharma' in another, with a different phone number because they moved. Entity resolution is the discipline of bringing those records together without a shared key, by comparing the fields you do have and scoring how likely it is that they are the same thing.

Imagine a busy railway station in a city like Mumbai, where a single person has bought tickets from three different counters, each run by a separate company. The first ticket says 'A. Sharma, Phone 98765 43210'. The second says 'Anita Sharma, Phone 98760 43210' — a typo in the last digit. The third says 'Anita S., Address: Malad West'. There is no ticket number shared across all three, so the system cannot just match on that. A human clerk at a single desk, if they had access to all three, could look at the name, the phone number, and the address, and say, 'These are clearly the same person, let me merge them.' But if the station handled a million passengers a day, no team of clerks could do this by eye. Entity resolution is the automated clerk. It looks at pairs of records, compares each field — name, phone, address — and assigns a similarity score to each comparison. Then it combines those field-level scores into an overall score, and decides whether the records match, are definitely different, or are too close to call and need a human to check. The analogy holds well because the clerk uses common sense, not an exact key. It breaks down in two crucial ways. First, a clerk can understand context: they know 'S.' is a common abbreviation for 'Sharma'. An automated system does not, unless you teach it. Second, a clerk can handle a handful of records; the station handles a million. The automated clerk must compare records without checking every possible pair, because a million records would mean five hundred billion comparisons, which would take too long. That is why blocking is necessary, which we will get to later. The analogy stops working when you ask the clerk to do the impossible: check every pair. Automated systems must be smarter about which pairs to look at.

Entity resolution is the process of determining when different records refer to the same real-world entity, in the absence of a shared unique identifier, by comparing the fields in the records and deciding whether they match.

This topic sits inside your Data Engineering note, which you already have. It connects directly to Data Contracts: a good data contract defines which fields must be populated for entity resolution to work, so the producer and consumer agree on the meaning of 'name' and 'address.' It also connects to the Semantic Layer, because entity resolution is often how you know that 'customer 42' in one system is 'customer 73' in another, so queries can be merged. It connects to Point-in-Time Data, because entity resolution can look ahead if you are not careful: if you merge two records because the address matches, but the address was updated later in one system, you are using future information. And it connects to Data Fabric and Mesh: a fabric must resolve entities across domains, while a mesh handles it locally per domain. From this page you will move to Data Pipelines, where entity resolution is often one stage in the pipeline, and eventually to MLOps, where the matching model is retrained as data changes.