Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 38 ยท Plugins and Coverage

A Tour of Third-Party Plugins

Meet the pytest plugins most teams rely on: random ordering to expose hidden dependencies, reruns for flaky tests, soft assertions, xdist for parallel runs, timeouts and more.

pytest has one of the richest plugin ecosystems of any test framework, with well over a thousand packages. You do not need most of them. A handful, however, solve problems every growing test suite meets: tests that secretly depend on each other, flaky tests, slow suites, hanging tests, and unreadable failure reports.

This lesson tours the plugins worth knowing, shows the ones that run in your browser for real, and describes the rest honestly as read-only. Installing a plugin is always pip install <name>; pytest finds it automatically, and pytest --version (or the header line of a run) lists what is loaded.

A few plugins need operating-system features the in-browser Python does not have. pytest-xdist needs worker processes and pytest-timeout needs threads or signals, so those two are shown as read-only commands. Everything else below was run for real.

Quick map

Plugin Solves Typical use
pytest-randomly Order-dependent tests Shuffles order, reseeds random every test
pytest-rerunfailures Flaky tests --reruns 2 retries failures
pytest-xdist Slow suites -n auto runs tests in parallel
pytest-timeout Hanging tests --timeout=30 fails a stuck test
pytest-check Many checks in one test Soft assertions that do not stop at the first failure
pytest-asyncio / anyio async def tests See testing async code
pytest-benchmark Performance regressions benchmark(func) fixture with statistics
pytest-sugar, pytest-clarity Readability Progress bar, nicer diffs
pytest-html Reports Self-contained HTML report
pytest-django, pytest-flask Frameworks Fixtures and settings for those frameworks
pytest-bdd Behaviour-driven style Gherkin scenarios as tests
hypothesis Property testing See Hypothesis
pytest-cov, pytest-mock Coverage, mocking See earlier lessons

pytest-randomly: expose hidden order dependence

Tests must pass in any order. If test B only passes because test A ran first and left some state behind, you have a bug in your suite. It stays hidden until someone runs a single test, reorders files, or enables parallel runs. pytest-randomly shuffles modules, classes and tests on every run (printing the seed so you can reproduce a failure with -p randomly --randomly-seed=1234), and reseeds random, numpy.random and Faker before each test.

Python Finding a hidden order dependency

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In file order both tests pass, which is why the bug survives for months. Some shuffled orders fail, and each failing seed is reproducible. The fix is to remove shared state: give each test its own list through a fixture or build it inside the test. Opt out of shuffling temporarily with -p no:randomly.

pytest-rerunfailures: handle flakiness honestly

A flaky test passes and fails without code changes: timing, ports, shared services. The right fix is to find the cause. Reruns are a stop-gap that keeps the build moving while you investigate, and the report still counts each rerun so the problem stays visible.

Python Marking one test as flaky

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The summary line reports the reruns. From the command line, pytest --reruns 2 --only-rerun ConnectionError retries every test but only for the named exception types, which avoids hiding real assertion failures. Avoid blanket reruns in your default configuration.

pytest-check: several checks, one test

A plain assert stops at its first failure, so you fix one thing, rerun, and find the next. With pytest-check, every check call is recorded and all failures are reported together, handy when validating a large object.

Python Soft assertions with check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Use soft assertions sparingly. A test that checks twenty unrelated things usually wants to be several tests, perhaps one parametrized test.

Read-only: parallel runs and timeouts

pytest-xdist distributes tests across CPU cores. It is the single biggest speed-up for large suites:

Terminal
pip install pytest-xdist
pytest -n auto              # one worker per CPU core
pytest -n 4 --dist loadscope  # keep tests of one module/class on the same worker

Parallel runs expose the same hidden dependencies as random order, plus shared resources: two workers must not use the same database file, port or temporary folder. Use tmp_path, per-worker names (the worker_id fixture), or a database per worker.

pytest-timeout fails a test that runs too long instead of letting CI hang for hours:

Terminal
pip install pytest-timeout
pytest --timeout=10                    # every test, 10 seconds
Python
@pytest.mark.timeout(2)
def test_must_be_fast():
    ...
INI
# pytest.ini
[pytest]
timeout = 30
timeout_method = thread      # or "signal" on Linux/macOS

Both commands need processes, threads or signals, which the browser sandbox does not offer, so run them on your own machine.

Choosing plugins wisely

flowchart TB
    Q{"Your problem"} -->|"Tests depend on order"| A["pytest-randomly"]
    Q -->|"Suite too slow"| B["pytest-xdist"]
    Q -->|"Test hangs CI"| C["pytest-timeout"]
    Q -->|"Flaky external service"| D["fix first, then rerunfailures"]
    Q -->|"Need to see all failures"| E["pytest-check"]

Match the plugin to a real problem; do not install them speculatively.

  • Pin plugin versions in requirements-dev.txt or pyproject.toml; a plugin update can change behaviour.
  • Declare required plugins in config with required_plugins = pytest-cov pytest-mock so a missing one fails early with a clear message.
  • Check that a plugin is maintained and supports your pytest version before you depend on it.
  • Run pytest --trace-config to see exactly which plugins loaded, and -p no:name to disable one while debugging.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is pytest-randomly useful?

    Shuffling reveals shared state between tests. The printed seed lets you reproduce any failing order.
  2. 2. What is the best use of pytest-rerunfailures?

    Reruns hide instability. Limit them to specific known-flaky tests or exception types and still fix the cause.
  3. 3. What does pytest -n auto do (with pytest-xdist)?

    -n auto starts one worker process per CPU core and spreads the tests across them.
  4. 4. How do you disable an auto-loaded plugin named randomly for one run?

    -p no:<name> blocks a plugin by its registered name for that run.

Summary

  • Install only plugins that solve a real problem: ordering, flakiness, speed, hangs, readability.
  • pytest-randomly exposes hidden dependencies; pytest-xdist multiplies speed but demands isolated tests.
  • pytest-rerunfailures and pytest-timeout keep CI moving, and neither replaces fixing root causes.
  • pytest-check reports all failed checks in one test.
  • Pin versions, use required_plugins, and debug with --trace-config and -p no:name.

Next: Project Layout and Configuration