Skip to content
Tutorials Center
My learning

Topics

Lesson 60 of 76 ยท Reproducibility and Production Habits

Testing Data Code and Data Validation

Test data science code: validate schema and value ranges, write unit tests for transforms with small fixtures, and check properties on random inputs.

Data code fails quietly. A function that computes revenue per customer will happily return a number even if half the rows were dropped by a bad join, a column arrived in cents instead of dollars, or a date was parsed in the wrong order. No exception, no warning, just a wrong chart in a report that someone will trust. Testing is how you turn those silent failures into loud ones.

There are two different things to test, and a good project does both. Code tests check that your functions do what you think they do, using small inputs you control. Data validation checks that the real data arriving today still looks the way your code assumes. This lesson builds both with plain Python and pandas, so every example runs right here. On your own machine you would run the tests with pytest; the pandas testing lesson covers its DataFrame comparison helpers.

flowchart LR
    A["Raw data"] --> B["Validate input<br/>(schema, ranges)"]
    B --> C["Transform code<br/>(unit tested)"]
    C --> D["Validate output<br/>(row counts, totals)"]
    D --> E["Report or model"]

Validation guards the data going in and out; unit tests guard the code in the middle.

Validate the data: schema first

A schema is the contract for a table: which columns exist, what type each has, and which values are allowed. Checking it first catches the cheapest and most common failures, such as a renamed column or a numeric field that became text.

A useful validator does not stop at the first problem. It collects every problem and reports them together, so you fix a bad file in one pass instead of ten.

Python A validator that collects every problem

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The bad file has a typo in a column name, a stray note ("about 20") that turned amount into text, and an extra column. All of it shows up at once. The contract uses small predicates such as is_float_dtype instead of exact dtype names, because the same column can legitimately be stored as object or str depending on your pandas version.

Validate the data: values and relationships

Types alone are not enough. An amount can be a float and still be negative, and an order_id can be an integer and still appear twice. Value checks encode what you know about the world:

  • Ranges: ages between 0 and 120, prices above zero, dates not in the future.
  • Allowed sets: a country code must be in a known list.
  • Uniqueness: an order ID identifies one row.
  • Completeness: key fields are never missing.
  • Relationships: a shipping date is not before the order date; line items add up to the order total.

Write each rule as a small named check that returns the offending rows. Returning the rows (not just True or False) makes the failure easy to investigate.

Python Rule-based value checks that return the bad rows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Not every failed rule deserves the same response. Decide up front which checks are errors (stop the pipeline, because results would be wrong) and which are warnings (log it and continue, because it is suspicious but plausible). An order of 8,000 might be a real bulk purchase; a negative amount is probably a refund that was coded wrongly, and you must decide what to do with it before computing revenue.

Tip: Validate at the boundaries. Check data when it enters your project and when it leaves a major step. Inside a step, trust the types and spend your effort on the code tests instead.

Dedicated libraries such as Pandera and Great Expectations package these ideas with nicer syntax and reports. They are worth learning, but they express exactly the rules you wrote above, so understanding the plain version makes any of them easy.

Unit tests for transforms

A transform is a function that takes a DataFrame and returns a new one: clean the names, compute a feature, aggregate by month. Tests for transforms follow one recipe:

  1. Build a tiny fixture: a DataFrame of three to six rows that you can check by eye.
  2. Write the expected result by hand.
  3. Run the function and compare.

Small fixtures beat samples of real data because you can see why the expected answer is right. Include the awkward cases on purpose: a missing value, a duplicate, an empty group, a boundary value.

Pytest finds functions named test_* and reports failures. Pyodide here has no pytest, so the next example uses a few lines of our own runner that does the same job.

Python Unit tests for a feature transform, with edge cases

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at what the tests pin down: the boundary (20.0 is small, 20.01 is medium), what happens to a missing value, and that the function does not modify its input. Those are the decisions a reader would otherwise have to guess. The last two tests also document behavior you chose on purpose.

Use assert_frame_equal rather than == to compare DataFrames: it checks values, dtypes, column order and index, and prints a clear difference when they do not match. For floating point results, it uses a tolerance, so 0.1 + 0.2 does not fail against 0.3.

What a failing test looks like

A test is only useful if it can fail. When you write a new one, break the code on purpose once and watch it go red. Here is the same kind of transform with a classic bug, and how a test exposes it:

Python A test that catches a classic join bug

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The inner join silently dropped customer 3's 40.0. The assert on totals is a reconciliation check: the output must add up to the input. These checks are cheap, work on real data, and catch a whole family of join and filter mistakes. Put them at the end of important steps.

Property checks on random inputs

Examples test the cases you thought of. Property checks test statements that must be true for any input, using many generated inputs. For a transform, good properties include:

  • The row count is unchanged (or changes in a known way).
  • Scaling then un-scaling returns the original values.
  • Rank order is preserved by a monotonic transform.
  • A normalized column has mean near 0 and standard deviation near 1.
  • A function applied twice gives the same result as once (idempotence).

Libraries like Hypothesis automate input generation, but a seeded loop gets you most of the value.

Python Property checks with seeded random inputs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Every property held in all 200 trials. If one had failed, you would print the failing input (with the seed, you can reproduce it exactly) and use it as a new fixture for an ordinary unit test. That is how property checks and example tests complement each other.

A caution: a constant column has a standard deviation of zero, so standardize would divide by zero and return missing values. The random inputs here never produce one. Finding such edge cases is exactly why you also write examples by hand.

Making testing a habit

  • Keep checks next to the code. A tests/ folder and a validate.py module, both in version control (see Reproducible Analysis).
  • Test what is risky. Joins, filters, date handling, grouping and anything with a manual rule deserve tests. A one-line column rename does not.
  • Fail loudly. A pipeline that continues after a failed check is worse than one that stops.
  • Fix, then pin. Every bug you find in real data becomes a fixture and a test, so it cannot return.
  • Do not test the libraries. You do not need a test proving that groupby sums correctly. Test your use of it.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the main difference between a unit test and data validation?

    Unit tests prove that functions behave correctly on fixtures you control. Validation checks that today's real data still matches the assumptions the code makes.
  2. 2. Why should a validator collect all problems instead of stopping at the first?

    A full report shows every defect at once, which makes diagnosing a bad delivery much quicker.
  3. 3. An inner join drops customers missing from a lookup table. Which check catches the lost revenue?

    A reconciliation check asserts that totals survive the step. Dropped rows show up as a lower total.
  4. 4. Why build test fixtures with only a few rows?

    With a handful of rows you can reason about the expected result and be sure it is right, including boundaries and missing values.
  5. 5. Which statement is a property that should hold for any input to a standardize function?

    Properties describe what must be true in general. Mean 0 and standard deviation 1 define standardization, as long as the input is not constant.

Summary

  • Silent data bugs are the main risk; tests and validation make them loud.
  • Validate the schema first (columns, dtypes), then values (ranges, allowed sets, uniqueness, relationships). Return the offending rows.
  • Decide which checks are errors that stop the pipeline and which are warnings.
  • Unit-test transforms with tiny, hand-checkable fixtures, including boundaries, missing values and "input not modified".
  • Use assert_frame_equal to compare DataFrames, and reconciliation checks (totals in equal totals out) after joins and filters.
  • Property checks on seeded random inputs find cases you did not think of; turn every bug into a permanent test.

Next: From Model to Decision and Deployment, where a trained model meets the real world.