Skip to content
Tutorials Center
My learning

Topics

Lesson 53 of 64 ยท Quality and Production

Testing and Validating DataFrames

Test pandas code with assert_frame_equal and assert_series_equal, tolerances, hand-written schema checks, small fixtures and property-style checks, plus a pytest example.

Data code is easy to break without noticing. You rename a column, change a merge, tweak a filter, and the script still runs. Whether the numbers are still right is a different question, and nobody asks it until a report looks odd.

Two habits answer that question. Tests check that your functions give the expected output for small, known inputs. Validation checks that real data coming into a pipeline has the shape and values you assumed. This lesson shows both using only what pandas ships with: the pandas.testing module, plain assert statements and a few small helper functions.

Comparing frames with assert_frame_equal

You cannot compare DataFrames with == in a test. It returns another DataFrame of booleans, and assert df1 == df2 raises an error about ambiguous truth values. pandas provides assert_frame_equal and assert_series_equal instead. They pass silently when the objects match and raise an AssertionError that names exactly what differs: values, dtype, index, column order, names or shape.

flowchart LR
    A["small input<br/>(fixture)"] --> B["function under test"]
    B --> C["actual DataFrame"]
    D["expected DataFrame<br/>(written by hand)"] --> E["assert_frame_equal"]
    C --> E
    E --> F["pass: silent"]
    E --> G["fail: AssertionError<br/>with the difference"]

A test builds a tiny input, runs your function, and compares the result with an expected frame you wrote by hand.

The defaults are strict about structure and dtype, but tolerant about floats (rtol=1e-5, atol=1e-8). The most useful switches are:

Argument Effect
check_dtype=False Allow int64 versus float64
check_like=True Ignore the order of rows and columns
check_exact=True Demand identical floats
rtol, atol Set the relative and absolute tolerance
check_names=False Ignore Series and index names
Python What assert_frame_equal catches

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that two NaN values in the same place count as equal here, which is what you want in tests, although NaN == NaN is false in Python.

Write tests around small fixtures

A fixture is the small, hand-made input a test uses. Keep fixtures to a handful of rows that show one idea each: a normal row, a duplicate, a missing value, an edge case. Write the expected output by hand, from the rules you mean to implement, never by running the function and pasting the result (that only proves the function equals itself).

The pattern is always the same: build input, call the function, compare. A function that takes and returns a DataFrame, with no hidden global state, is trivial to test, which is one reason pipelines are built from such functions.

Here is a tiny cleaning function and two tests for it. The tests are ordinary functions whose names start with test_, the convention pytest uses. A short loop below plays the role of the test runner, so you can run it right here:

Python Two tests and a tiny runner

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The third test is worth copying into your own suite. Pipelines that modify their input in place cause the nastiest bugs, and one test per function is a cheap guard.

The same tests with pytest

On your own machine you would put those functions in a file called test_pipeline.py and run pytest. This block is read-only because the browser cannot run pytest:

Python
# test_pipeline.py
import pandas as pd
import pytest
from pandas.testing import assert_frame_equal

from pipeline import add_total


@pytest.fixture
def orders():
    return pd.DataFrame({"price": [2.0, None, 5.0], "qty": [3, 1, 4]})


def test_rows_without_price_are_dropped(orders):
    assert add_total(orders)["qty"].tolist() == [3, 4]


@pytest.mark.parametrize("qty, expected", [(0, 0.0), (3, 6.0)])
def test_total_for_quantity(qty, expected):
    df = pd.DataFrame({"price": [2.0], "qty": [qty]})
    assert add_total(df)["total"].iloc[0] == pytest.approx(expected)

pytest.fixture shares a fixture between tests, parametrize repeats a test over several inputs, and pytest.approx is pytest's own float tolerance. Everything else is the pandas.testing you already used.

Validating data by hand

Tests protect your code; validation protects you from the data. A schema check answers: are the columns I need present, are the dtypes right, are values in range, are keys unique? You do not need a library to start. A function that returns a list of problems is easy to read, easy to test and easy to print:

Python A hand-written schema check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When this runs on a real feed, decide the policy up front: stop the whole job on a serious problem (missing columns, wrong dtypes), but only warn and quarantine rows for small ones. Never delete bad rows silently.

Assertions inside a pipeline

Inside a pipeline, use plain assert statements as tripwires at the boundaries between steps. They document your assumptions and fail early, close to the cause. Typical checks are row counts after a merge, unique keys and no unexpected missing values:

Python
merged = orders.merge(customers, on="customer_id", how="left", validate="m:1")
assert len(merged) == len(orders), "merge changed the row count"
assert merged["segment"].notna().all(), "orders without a matching customer"

The validate="m:1" argument makes merge itself raise if the right side has duplicate keys, which is a classic way to multiply rows. One caution: Python skips assert when it runs with -O. For checks that must always run in production, raise an explicit exception such as ValueError instead, which is what the helper below does.

Property-style checks

A property is a statement that must hold for any input, not just your hand-picked rows. You generate many random inputs with a fixed seed and check the property on each. You are no longer asking "does this output equal that output?" but "is the output always consistent?".

Good properties for data code:

  • Group totals add up to the overall total.
  • Sorting returns the same rows (same multiset), just reordered.
  • A function applied twice gives the same result as applied once (idempotent).
  • A merge on a unique key never changes the row count.
Python Properties on seeded random data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If you use the Hypothesis library on your own machine, it generates the inputs for you (including nasty edge cases) and shrinks failures to a minimal example. The idea is the same as the loop above.

What to test and what not to

  • Test your logic: filters, joins, derived columns, edge cases like empty frames, all-missing columns and duplicate keys.
  • Do not test pandas itself. df.groupby("a").sum() works; your choice of key and column is what could be wrong.
  • Keep tests fast: a few rows, no network, no huge files.
  • Add a test every time you fix a bug, using the row that broke production as the fixture.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why use assert_frame_equal instead of assert df1 == df2?

    df1 == df2 gives a frame of booleans, so assert raises an ambiguous-truth-value error. assert_frame_equal returns nothing on success and a detailed AssertionError on failure.
  2. 2. Two frames are identical except one has a column as int64 and the other as float64. What does assert_frame_equal do by default?

    dtypes are checked by default. Pass check_dtype=False when only the values matter.
  3. 3. Where should the expected DataFrame in a test come from?

    If you paste the function's own output as the expectation, the test can never fail. Work out the expected values independently.
  4. 4. Which is a property that should hold for a correct groupby-sum report?

    Totals must be conserved. A mismatch points to rows dropped by missing keys or a wrong join.
  5. 5. Why might a production check raise ValueError instead of using assert?

    Optimised mode removes assert statements, so a check that must always run should raise an explicit exception.

Summary

  • Compare frames with pandas.testing.assert_frame_equal and assert_series_equal; they report the exact difference.
  • Floats are compared with a tolerance by default; tune rtol, atol, check_dtype, check_like and check_exact.
  • Build tiny fixtures, write expected output by hand, and test that functions do not modify their input.
  • Validate incoming data with a function that returns a list of problems: columns, dtypes, uniqueness, ranges, missing values.
  • Use assertions (or explicit exceptions) at step boundaries, and validate= in merge.
  • Property-style checks on seeded random data verify invariants such as conserved totals and idempotence.
  • pytest adds fixtures, parametrize and approx on top of the same ideas.

Next: Building Data Pipelines