Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 38 ยท Plugins and Coverage

Property-Based Testing with Hypothesis

Let the Hypothesis library generate hundreds of inputs for your pytest tests, state properties instead of examples, and watch failing cases shrink to the smallest counterexample.

Every test you have written so far is example-based: you pick an input, compute the expected output, and compare. That is great, but you only test the cases you thought of, and bugs love the cases you did not think of: empty strings, negative numbers, huge values, odd Unicode.

Property-based testing flips the approach. You describe what must always be true (a property), and a library generates many inputs, looking for one that breaks the property. The standard library for this in Python is Hypothesis, and it plugs into pytest with a decorator.

Your first property

Instead of "reversing [1, 2, 3] gives [3, 2, 1]", state a rule: reversing a list twice gives back the original list, for any list of integers.

Python A property that always holds

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

@given(strategy) tells Hypothesis how to build the arguments. st.lists(st.integers()) produces lists of every shape: empty, one element, long, with negatives and huge numbers. Each test body runs up to 200 times. (database=None and derandomize=True just make the demos reproducible and leave no files behind; in your own project you normally keep the defaults.)

Strategies

Strategies are the building blocks. They compose, so you can describe nearly any data:

Strategy Generates
st.integers(min_value=0, max_value=100) whole numbers in a range
st.floats(allow_nan=False) floats, with NaN and infinities optional
st.text() / st.text(alphabet="abc") strings, including odd Unicode
st.booleans(), st.none() True/False, None
st.lists(s, min_size=1) lists of whatever s makes
st.dictionaries(keys, values) dicts
st.tuples(a, b) fixed-shape tuples
st.sampled_from([...]) one of a fixed set
st.one_of(a, b) either strategy
st.builds(Cls, x=..., y=...) instances of your own class

Transform values with .map() and .filter() (filter sparingly: if most values are rejected, generation becomes slow), or combine with st.composite for dependent values.

Shrinking: the best feature

When Hypothesis finds a failing input it does not hand you the first huge random one. It shrinks it: repeatedly simplifies the input while it still fails, until it reaches a minimal counterexample. Here is a deliberately buggy function:

Python Finding and shrinking a bug

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The report names the smallest failing input, something like x=0, low=-1, high=0. A hand-written example test would probably never have used zero together with a negative lower bound. The shrunk example is also a perfect regression test: copy it into an ordinary example test with @example(...) or a plain test function.

flowchart LR
    G["generate random input"] --> R["run the test"]
    R -->|passes| G
    R -->|fails| S["shrink: try simpler inputs"]
    S -->|still fails| S
    S -->|smallest failing| P["report counterexample"]

Generation, execution and shrinking form a loop that ends with a minimal failing case.

What properties should I check?

Finding properties is the skill. A few reliable families:

  • Round trip: decode(encode(x)) == x (serialization, parsing, compression).
  • Invariants: the output is always sorted, within bounds, unique, non-negative.
  • Idempotence: doing it twice equals doing it once (normalize(normalize(x)) == normalize(x)).
  • Compare with a simple model: your fast implementation equals an obviously correct slow one.
  • No crash: for any input, the function either returns a valid result or raises a documented error type.
  • Metamorphic: changing the input in a known way changes the output predictably (f(x + 1) == f(x) + 1).
Python Round trip and idempotence

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

@example(...) adds a specific case that always runs, assume(condition) discards inputs you do not care about (prefer a narrower strategy over many assume calls).

Practical advice

  • Keep both styles. Example tests document behaviour; property tests hunt for surprises.
  • Control cost. Tune with @settings(max_examples=..., deadline=...), or use profiles registered in conftest.py so CI runs more examples than your laptop.
  • Failures are saved. Hypothesis keeps failing inputs in a .hypothesis directory and replays them first on the next run; commit nothing from it, but do not delete it mid-debugging.
  • Mixing with fixtures: function-scoped fixtures are not reset between generated examples, which triggers a health check. Use st.data(), build the object inside the test, or use a wider-scoped fixture deliberately.
  • Stateful testing (RuleBasedStateMachine) can drive whole APIs with random sequences of operations, a strong technique for caches, queues and databases.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is a property in property-based testing?

    A property is a rule that must be true for every input in a described set. Hypothesis then tries to find a counterexample.
  2. 2. What does shrinking do?

    After finding any failure, Hypothesis searches for the simplest input that still fails, which makes the bug easy to understand.
  3. 3. Which is a round-trip property?

    Round-trip properties check that decoding an encoding returns the original value.
  4. 4. What is @example(...) for?

    @example pins a concrete case, such as a past bug, so it is checked on every run.

Summary

  • Hypothesis generates inputs from strategies and checks that a property holds for all of them.
  • Failing inputs are shrunk to minimal counterexamples and remembered for the next run.
  • Good properties: round trips, invariants, idempotence, model comparison, "never crashes", metamorphic rules.
  • Use @example for known tricky cases and assume sparingly.
  • Combine property tests with ordinary example tests instead of replacing them.

Next: conftest.py Hooks and Local Plugins