11. Data quality, contracts, lineage and observability

Data fails silently: a broken pipeline usually runs green and produces plausible-looking wrong output. Trust therefore has to be engineered: define what correct means, test it at every boundary, make producers commit to it, and monitor data the way site reliability engineers monitor services.

Dimensions of quality

Dimension	Question it answers	Example check
Completeness	Is everything there?	Today’s row count within 20% of the 7-day average
Validity	Does each value obey its rules?	Age between 0 and 120; country code in the ISO list
Uniqueness	Is each entity recorded once?	No duplicate request_id
Consistency	Do related datasets agree?	Every order references an existing customer
Accuracy	Does it match reality?	Sampled records verified against the source system
Freshness	Is it recent enough?	Latest partition is less than 2 hours old
Representativeness	Does it cover the cases the model will meet?	Share of each language or customer segment against production traffic
Label quality	Are the targets right?	Agreement between two annotators on a shared sample

The last two matter most for AI and are the ones classic data teams tend to skip.

Three layers of tests

1. Schema tests: types, nullability, allowed values. Cheap; run them everywhere.
2. Row-level assertions: ranges, formats, referential integrity between tables.
3. Dataset-level statistical checks: row counts, null rates, distinct counts, and distribution shift compared with a reference.

Give every test a severity. Some failures must block publishing; others should warn a human. If everything blocks, people disable the tests; if everything warns, nobody reads the warnings.

Tools: dbt tests for SQL models, Great Expectations and Soda for general checks, Pandera for DataFrames, Pydantic for validating individual records in Python, and Deequ for Spark.

Where to test

• At ingestion: reject or quarantine data that breaks the contract before it spreads.
• After each transformation: check invariants, such as row counts before and after a join.
• Before publishing: the audit step of write-audit-publish (section 10).
• At consumption: validate model inputs at training and inference time, so a bad feature never reaches the model unnoticed.

Data contracts

Most data breakage starts upstream: a product engineer renames a column without knowing a model depends on it. A data contract is an explicit agreement between a producer and its consumers, covering:

• The schema and the meaning of each field.
• Guarantees: freshness, completeness, allowed values.
• An owner and a contact.
• How changes are proposed, versioned and announced.

The key move is shifting left: enforce the contract in the producer’s CI pipeline, for example with schema compatibility checks against a schema registry. A breaking change then fails the producer’s build instead of silently breaking a model weeks later.

Lineage

Lineage is the graph of how every dataset is derived from others, at table level and ideally at column level. It answers questions that are otherwise unanswerable:

• Impact: if I change this column, which tables, dashboards and models break?
• Root cause: this number is wrong, so which upstream step caused it?
• Compliance: where does this personal data flow, and which models were trained on data from a user who asked to be deleted?

OpenLineage is the open standard for emitting lineage events from pipelines. DataHub, OpenMetadata, Marquez and Unity Catalog collect and display it, alongside catalog metadata such as owners, descriptions and PII tags.

Data observability

Observability means continuously watching data health signals, not only running fixed tests:

• Freshness: did the data arrive on time?
• Volume: is the amount normal?
• Schema: did columns appear, disappear or change type?
• Distribution: did values shift: null rate, mean, category mix?
• Lineage: what is affected downstream of an anomaly?

Anomaly detection on these metrics catches problems no one thought to write a test for. Route each alert to the dataset’s owner, and tune thresholds ruthlessly; alert fatigue is how real incidents get ignored.

Handling a data incident

1. Detect, ideally before a consumer does.
2. Contain: pause downstream jobs so bad data stops spreading, and tell consumers.
3. Fix the code or the source.
4. Backfill the affected partitions, which only works if pipelines are idempotent.
5. Review: write a short postmortem and add the test that would have caught it.

Quality for machine learning

• Label noise: measure agreement between annotators (Cohen’s kappa is a standard statistic), seed tasks with known-answer “gold” items, and use tools such as cleanlab to flag likely mislabeled examples.
• Drift: data drift is a change in the input distribution; concept drift is a change in the relationship between inputs and the target. Both degrade models after deployment. Evidently and similar tools compute drift reports.
• Documentation: datasheets for datasets and data cards record where data came from, how it was collected, its known gaps and intended uses.
• LLM data checks: language identification, quality and toxicity scores, PII detection, near-duplicate rates, and benchmark contamination checks. Section 13 covers these in depth.

Questions to ask

• What does “correct” mean for this dataset, written down?
• Which failures block publishing, and which only warn?
• Who is told when the schema changes upstream, and before or after it ships?
• If this column is wrong, which models and dashboards are affected?
• Does the training data still represent what the model sees in production?