Data Contracts

Imagine you are working on a project where one team sends data to another team's dashboard or automated report. Without a clear agreement on what that data contains, the receiving team constantly has to guess. They might assume a number is there when it is actually missing, or interpret a date format differently. This leads to broken reports and mistrust between teams. A data contract removes this guessing game. It ensures that the team sending the information and the team using it share the same understanding of what the data means and how it looks. This allows you to change one part of your system without accidentally breaking everyone else's work, saving time and preventing costly errors in downstream processes like automated machine learning models or large-scale data processing.

Consider a simple agreement between two people sharing a recipe book. Before cooking, you must agree on the meaning of specific terms. For instance, when a recipe says 'flour', it clearly means wheat flour, not besan. When it says '1 cup', it refers to the same standard measuring cup for both of you. If these agreements do not exist, your friend might add sugar when you meant salt, or use a different-sized cup, and the resulting dish will be wrong. A data contract functions similarly but between computer systems. One system (the producer) sends data, such as information from a sensor or a database table. Another system (the consumer), like a dashboard or an automated model, receives and uses that data. The contract explicitly states what the data looks like (the structure of fields and their types), what each field means (the specific definition or rule for that data), how accurate it must be, and how changes to this structure are handled. Without this agreement, the consumer might assume a column contains numbers but receive text instead, or assume a value is always positive when it can be negative. This results in incorrect answers from models or reports. The contract ensures both sides agree on these details before any data is processed.

Think of a bus timetable shared between the bus company and a passenger. The bus company promises that a specific bus will arrive at 8:05 AM at this stop, going to the city centre. The passenger relies on this precise promise to catch their connection to work. If the bus arrives at 8:04, the passenger might miss it; if it arrives at 8:06, they are late. The timetable acts as a contract: it defines the exact arrival time (the structure of the data), the specific route and destination (the meaning of the data), the reliability standard (how often the bus operates on time), and the process for updates (new timetables are published weeks in advance, not changed silently). This analogy shows how contracts set expectations and reduce surprise. However, there is a key difference: a bus timetable is read by humans who can tolerate small ambiguities or wait if the bus is slightly late. A data contract is read by computer systems, which follow rules literally. If a system expects a number but receives text, it will either crash or produce garbage results immediately. Therefore, the contract must be precise and understandable to machines, not just people. Additionally, while bus timetables remain static for months, data structures may change frequently as new features are added, requiring a clear protocol for how both sides adapt without breaking.

A formal agreement between a system that generates data and a system that uses it, which clearly defines the structure of the data fields, the specific meaning of each field, the required quality standards, and the rules for how changes to this structure will be managed and communicated.

This concept is the starting point for understanding how different parts of a data system communicate. It directly relates to the 'Semantic Layer', which is the component that defines exactly what business metrics mean by providing a shared understanding of data definitions across an organization. It also connects to 'Data Fabric and Mesh', architectures that treat data as a product owned by specific teams, where the interface between these teams is defined by such contracts. Finally, it relates to how data is stored in efficient formats like Parquet files, which embed structural information about the data columns directly within the file to help systems read and process the information correctly.