Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 41 ยท CI/CD for ML

Data Validation in CI

Stop bad data before it reaches training: write schema, range, null, category and freshness checks with pandas and fail the pipeline with a clear report.

Most production ML incidents are not clever model failures. They are boring data failures: a column renamed upstream, prices suddenly in cents instead of dollars, a join that silently dropped half the rows, a category that appeared for the first time. The model happily trains on that data and produces confident nonsense.

Data validation is the habit of writing down what good data looks like and checking every new batch against it before training or serving. In this lesson you will build a small validator with pandas, learn the main kinds of checks, and see how to turn the result into a pass or fail for your pipeline.

The five families of checks

What to check

Schema Columns exist with the right types.
Ranges Values stay in a sane interval.
Missing values Null share below a limit.
Categories Only known labels appear.

Plus row counts and freshness.

1 / 2
  1. Schema: required columns, data types, no unexpected extra columns.
  2. Range and domain: age between 0 and 120, price positive, probability between 0 and 1.
  3. Completeness: the fraction of nulls in each column stays below a limit.
  4. Categorical values: a country column contains only known codes (and you decide whether an unknown code is an error or just a warning).
  5. Volume and freshness: a daily batch usually has roughly 50,000 rows and the newest timestamp is no more than a day old.

A hand-written validator

Libraries such as Great Expectations, Pandera and TFX Data Validation do this at scale (see the tooling landscape). Before using one, write a minimal version yourself so you know what it does. The design is: each check returns a small result object, and a runner collects them all rather than stopping at the first problem. Seeing every problem at once saves many reruns.

Python Schema, range and null checks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first batch has one null income out of three rows, which is 33 percent, well above the 5 percent limit. Check the printed report to see whether the validator agrees with you. That is a useful habit: always run a validator on both a known-good and a known-bad sample, or you may only be testing that it never complains.

Volume, freshness and duplicates

A batch can satisfy every column rule and still be wrong: it may be empty, a week old or contain the same rows twice. Add dataset-level checks.

Python Row count, freshness and duplicates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three problems appear: only 5 rows where about 100 are expected, data 48 hours old, and a duplicated order id. A pipeline with only column-level rules would have passed all three.

Turning checks into a pipeline gate

Validation is only useful if it can stop something. Wrap the validator in a script whose exit code reflects the result. In CI this is one more step; the next step (training) does not run if it fails.

Python
# scripts/validate_data.py
import sys
import pandas as pd
from validation import validate, SCHEMA

df = pd.read_parquet("data/latest.parquet")
problems = validate(df, SCHEMA)
for p in problems:
    print("DATA CHECK FAILED:", p)
sys.exit(1 if problems else 0)

Think about severity too. Some issues must block (missing label column). Others should only warn (a new category that might be harmless). A simple convention is two lists, errors and warnings, where only errors set a non-zero exit code. Warnings should still be recorded so you can see trends.

Learn the rules from data

Writing every range by hand does not scale to hundreds of columns. Tools can infer a starting schema from a trusted dataset (min, max, null share, categories), which you then review and tighten. A sketch of the same idea:

Python Infer a schema from trusted data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Always review inferred rules. A margin of 10 percent around the range seen last month would reject a perfectly legitimate 66-year-old customer. Inferred schemas are a draft, not a verdict.

Common mistakes

  • Validating only training data. Validate serving inputs too; they break independently.
  • Stopping at the first error. Report all problems in one run.
  • Thresholds that never fail. Rules loosened until nothing ever triggers are no rules at all.
  • Ignoring the label. Check the target column's range and class balance too; label bugs are expensive.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A batch has every column in range but only 5 rows instead of about 100,000. Which kind of check catches it?

    Volume (row count) checks look at the dataset as a whole, not at individual column values.
  2. 2. Why should a validator collect every problem instead of raising on the first?

    A complete report saves repeated pipeline runs.
  3. 3. How does a CI step stop the pipeline when data is bad?

    The exit status controls whether later steps run.
  4. 4. What is the risk of an inferred schema used without review?

    Inferred ranges reflect past data only. A human should review and adjust them.

Summary

  • Validate schema, ranges, null share, categories, row counts, freshness and duplicates.
  • Collect all problems into one report, and separate blocking errors from warnings.
  • Test your validator on known-good and known-bad data.
  • Run validation on ingestion, before training and at serving time.
  • Inferred schemas are a starting point that people must review.

Next: Model Quality Gates