Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 36 ยท Orchestration and Quality

Data Quality: Checks, Contracts and Quarantine

Build data quality into pipelines: the dimensions of quality, row-level and dataset-level checks, freshness, anomaly detection, data contracts and quarantining bad rows.

A pipeline that runs green but produces wrong numbers is worse than one that fails loudly. Dashboards lose trust the first time a manager spots an impossible figure, and trust is slow to rebuild. Data quality is the practice of defining what "good" data looks like, checking it automatically, and deciding what to do when it is not.

This lesson builds a small validation framework with pandas, then maps it to the tools used in real teams (dbt tests, Great Expectations, Soda, Deequ).

Dimensions of quality

Dimension Question Example check
Completeness Is anything missing? email not null for 99%+ of rows
Uniqueness Any duplicates? order_id unique
Validity Is it in the allowed form? status in (paid, shipped, returned)
Accuracy Does it match reality? Totals match the source system
Consistency Do tables agree? Every customer_id in orders exists in customers
Timeliness Is it fresh enough? Latest event under 2 hours old
Volume Is the quantity plausible? Row count within 30% of last week

Row-level checks

Python Validate rows and split good from bad

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The bad rows are not dropped silently. They go to a quarantine (dead-letter) table with the reason, so someone can fix the source or the rule and replay them.

Dataset-level checks

Some problems only show in aggregate: the table is empty, row counts fell by half, a column's average jumped 10x, or yesterday's partition never arrived.

Python Volume, freshness and drift checks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Fixed thresholds work for known limits (freshness, allowed values). For volumes that vary with weekday and season, compare with the same weekday last week or use a rolling baseline.

A tiny reusable framework

Wrapping checks as named functions with a severity makes them composable:

Python Stand-in: rule runner with error vs warn severity

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The useful distinction: errors stop the pipeline (the table is unusable), warnings alert (look into it, but publishing is still better than nothing). Stopping too eagerly causes alert fatigue and missing data; never stopping hides problems.

Where to put checks

flowchart LR
  S["Source"] --> I["Ingest"]
  I -->|"schema + volume + freshness"| R["Raw"]
  R --> T["Transform"]
  T -->|"keys, nulls, referential, business rules"| M["Marts"]
  M -->|"totals reconcile with source"| D["Dashboards"]
  • At ingestion: schema matches the contract, file not empty, freshness.
  • After transforms: unique keys, not-null, accepted values, relationships.
  • Before publishing: reconciliation against a source of truth (sum of invoices equals ledger).

Data contracts

A data contract is an agreement between a data producer and its consumers: schema, types, meaning, freshness and quality guarantees, owner and a change process. Instead of finding out about a renamed column when dashboards break, the producer's CI checks the contract and blocks the breaking change.

Python Check an incoming batch against a contract

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Tools you will meet

  • dbt tests: unique, not_null, accepted_values, relationships, plus custom SQL tests.
  • Great Expectations / Soda / Deequ: declarative "expectations" with reports and thresholds.
  • Warehouse features: constraints (informational in many warehouses), table expectations in Delta Live Tables.
  • Observability tools (next lesson): monitor freshness, volume and schema drift automatically.

Practical advice: start with a few checks that catch the failures you have actually had (duplicates, nulls in keys, late data), and grow from incidents. A long list of checks nobody reads is not quality.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why quarantine bad rows instead of dropping them?

    Quarantine with a reason gives visibility and a path to recover data.
  2. 2. Which check detects that a daily partition never arrived?

    Freshness and volume checks catch missing or abnormally small data.
  3. 3. What is a data contract?

    Contracts make expectations explicit so breaking changes are caught before consumers break.
  4. 4. What is the difference between error and warning severity?

    Severity lets you stop on unusable data while merely flagging softer problems.

Summary

  • Quality has dimensions: completeness, uniqueness, validity, accuracy, consistency, timeliness, volume.
  • Check rows and datasets; quarantine failures with reasons; use severity to decide whether to block.
  • Place checks at ingestion, after transformation and before publishing; use contracts with producers.

Next: Testing, Monitoring and Observability