Found Data

Found data is the raw material of everything you will build. Every model you train, every dashboard you make, every conclusion you draw starts from data you did not design. Learning to see the difference between data collected for a purpose and data found lying around is what lets you judge whether your analysis means anything at all. This unlocks further study in Data Quality, Data Cleaning, Data Validation, and eventually Model Monitoring. Without this concept, every later decision about missing values, outliers, and constraints has no foundation.

Think of the difference between a diary written for yourself and an email you wrote to a friend. The diary is designed to record your thoughts; the email was written to communicate something else, but you can still read it later to learn what you were worried about. Found data is like the email: it was never created to be analysed. It was created to run a business, to sell a product, to log a transaction, to post on social media. But someone comes along later and says, 'These emails tell me about worry patterns,' or 'These sales records tell me about customer habits.' The data was found, not designed for that question. That origin leaves a mark on everything you do with it. The person who wrote the email had no idea you would read it for anxiety signals, so they did not write their true feelings clearly; they wrote what they wanted the recipient to see. Similarly, the salesperson who logged a discount did not know you would later use that to predict demand; they logged what the software required. The data carries the fingerprints of its original purpose, and you have to learn to see those fingerprints before you trust anything it tells you.

Imagine a city's surveillance cameras. They were installed to catch traffic violations, to monitor crowds, to deter crime. Nobody installed them to study how people move through the city at different times of day. But a city planner comes along and says, 'I can use these camera feeds to understand pedestrian flow and design better sidewalks.' This is found data. The cameras now serve a second purpose they were never designed for. The crucial thing: the way the cameras were placed biases what they see. They are dense in commercial areas, sparse in residential zones, angled at intersections rather than mid-block. A planner who forgets this will conclude that no one walks in residential areas — but that is an artefact of camera placement, not a fact about walking. The same happens in data. A social media platform logs every click on the 'like' button but not the time someone hovers over a post without clicking. If you study engagement using only likes, you miss all the silent readers. The log was built for the platform's needs, not yours. Where the analogy breaks down: a camera has a clear field of view, but data has no obvious boundary. You cannot always tell where the data stops being representative and starts being distorted. The camera at least shows you what it sees; a dataset hides what it does not contain. That invisible bias is the harder problem.

Found data is data that was generated for another purpose — running a business, logging a transaction, serving a product — and is repurposed for analysis, carrying the biases of its original collection process into every conclusion drawn from it.

You have not studied Data Quality yet, but found data is the starting point of that entire subject. When you move on to Data Cleaning, you will see that the mess in real datasets — missing values, duplicates, outliers — is often a direct consequence of data being found rather than designed. The pattern of what is missing, for instance, is itself information about how the data was collected. Data Dictionaries will become your tool for documenting where each field came from and what its original purpose was. And Data Validation Constraints will be the way you encode the expectations that found data so often violates. For now, the key connection to hold: every other topic in Data Quality is a response to the fact that data was not made for you.