Data Contracts

This unlocks everything downstream in your data engineering path: building data pipelines, MLOps, and big data processing. Without a data contract, every pipeline is a fragile guess about what the data looks like. With one, you can change a schema without breaking every consumer, and you can trust the semantic layer on top. This is the foundation for trust between systems — without it, every other system you build is built on sand.

You and a friend decide to share a recipe book. Before you start cooking, you agree on what each ingredient means: 'flour' means wheat flour, not besan; 'milk' means cow's milk, not almond milk; '1 cup' means the same measuring cup. If you don't agree, your friend might add sugar when you meant salt, or use a different-sized cup, and the dish turns out wrong. A data contract is the same agreement, but between two computer systems: a producer (the system that sends data, like a sensor or a database) and a consumer (the system that receives and uses it, like a dashboard or a model). The contract spells out what the data looks like (schema), what each field means (semantics), how good the data must be (quality), and how changes are made (change process). Without it, the consumer might assume a field is a number when it's actually text, or assume a value is always positive when it can be negative — and the model or report built on top gives wrong answers. A contract makes the producer and consumer agree before anything breaks.

Think of a bus timetable. The bus company (producer) and the passenger (consumer) agree on a schedule: the bus will arrive at 8:05 AM, at this stop, going to the city centre. The passenger depends on that promise to get to work on time. If the bus arrives at 8:04, the passenger might miss it; if it arrives at 8:06, the passenger is late. The timetable is a contract: it specifies the arrival time (schema), the route (semantics), the reliability (quality — the bus shouldn't be broken down too often), and how changes are announced (change process — a new timetable is published weeks in advance, not silently changed on the day). The timetable teaches what a contract does: it sets expectations, reduces surprise, and ensures trust. But the analogy breaks down: a bus timetable is read by humans, who can handle ambiguity — if the bus is a minute late, they wait a bit. A data contract is read by computer systems, which are literal: if the data says 'age' is a number, but the consumer gets a text string, it errors out or silently writes garbage. So the contract has to be precise and machine-readable, not a human-friendly document. Also, a bus timetable is static for months, but data schemas might change every week as new features are added. The contract has to include a change process that lets both sides adapt without breaking the system.

An explicit agreement between a data producer and a data consumer that specifies the schema (what fields, what types), the semantics (what each field means), the quality (how good the data must be, like completeness and accuracy), and the change process (how and when the schema can evolve, so the consumer isn't surprised).

You haven't studied data engineering topics yet, but this concept sits at the very start of the path. It connects directly to the 'Semantic Layer' — the shared definitions of what metrics mean — because you can't have a meaningful semantic layer if the data underneath it is different everywhere. It also connects to 'Data Fabric and Mesh', where mesh treats data as a product with an owning team and an interface — that interface is exactly a data contract. And it connects to 'Parquet and Flat-File Hygiene', because embedded schema in columnar files is a technical way of expressing part of a contract. For now, just know this is the foundation: everything later assumes contracts exist.