Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 36 ยท Orchestration and Quality

Testing, Monitoring and Observability for Pipelines

Test transformations with unit and integration tests, log run metrics, monitor freshness, volume and schema drift, and respond to incidents with alerts and runbooks.

Data code is code. It has bugs, it changes, and many people depend on it. But data pipelines have an extra problem: the code can be perfectly correct while the data changes underneath it (a source adds a column, a partner sends yesterday's file twice). So you need two defences: tests that prove your logic is right before release, and monitoring that notices when reality drifts after release.

Test layers for data work

Layer What it checks Typical tool
Unit tests One transformation function with small hand-made input pytest
SQL/model tests A model on fixture data gives expected rows dbt unit tests, pytest + DuckDB
Data tests Properties of real data (unique, not null) dbt tests, expectations
Integration tests The pipeline end-to-end on a small sample CI with test containers
Reconciliation Output totals match the source of truth scheduled checks

The cheapest, most valuable habit: write transformations as pure functions (data in, data out) so they are easy to test without a cluster.

Unit-testing a transformation

Python A pure function and its tests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Edge cases deserve tests: empty input, nulls, duplicates, timezone boundaries, a single row, and the same run twice (idempotency).

Testing SQL logic

You can test SQL on a tiny in-memory database seeded with fixtures:

Python Test a SQL model with fixtures

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Metrics from every run

A run that "succeeded" can still be wrong. Record metrics per run so you have a baseline: rows read, rows written, rows rejected, duration, bytes, max event timestamp.

Python Log structured run metrics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Write these as structured (JSON) logs or into a pipeline_runs table, and alert on rate of change: reject rate above 5%, rows written 40% lower than last week, max event time older than the SLA.

Data observability

Data observability monitors the health of data itself, borrowing from application observability. Common pillars:

  • Freshness: when did this table last update?
  • Volume: is the row count normal?
  • Schema: did columns appear, vanish or change type?
  • Distribution: did the null rate, mean or distinct count shift?
  • Lineage: which upstream tables and downstream dashboards are affected?
Python Detect schema drift between two runs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Alerts that people act on

flowchart LR
  M["Metric or check"] --> A{"Alert rule"}
  A -->|"severe"| P["Page on-call"]
  A -->|"moderate"| T["Ticket / channel"]
  A -->|"info"| D["Dashboard only"]

Good alerts name the dataset, the check, the expected versus actual value, the run id and a link to the runbook. Bad alerts say "Task failed". Route by severity so that on-call engineers are paged for broken business-critical data, not every warning, or they will start ignoring pages.

Incidents and runbooks

  1. Detect: alert fires (or a user reports a bad number).
  2. Contain: stop downstream publishing or flag the dataset as stale/untrusted.
  3. Diagnose: use lineage and run metrics to find the first bad step.
  4. Fix and backfill: correct the logic, rerun idempotently (see the earlier lesson).
  5. Review: write a short post-mortem and add the missing test or check.

Keep a runbook per critical pipeline: what it does, owners, common failures, how to rerun, and who consumes it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why write transformations as pure functions?

    Data in, data out makes tests trivial and repeatable.
  2. 2. A pipeline run succeeds but writes 60% fewer rows than usual. What catches this?

    Per-run metrics compared to a baseline reveal abnormal volume even when no task fails.
  3. 3. Which is a breaking schema change for consumers?

    Removing or retyping a column breaks queries that depend on it; additive columns usually do not.
  4. 4. What makes an alert useful?

    Actionable context and sensible routing prevent alert fatigue.

Summary

  • Test pure transformations, SQL models and end-to-end flows; include edge cases and idempotency.
  • Record run metrics and monitor freshness, volume, schema and distributions.
  • Alert on actionable thresholds and keep runbooks; after every incident add a test or a check.

Next: Data Catalogs, Metadata and Lineage