Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 38 ยท CI and Best Practices

Faster Suites and Flaky Tests

Find slow tests with --durations, run only what you need with -k, -m and --lf, share expensive setup with fixture scopes, and track down the real causes of flaky tests.

A test suite is only useful if people run it. When it takes twenty minutes, developers run it less, merge with fingers crossed and start ignoring red builds. When some tests fail at random, trust collapses faster still. Speed and reliability are therefore not luxuries: they decide whether the suite protects you or merely decorates the repository.

This lesson gives you a toolbox for both problems: measure first, select tests smartly, cut setup costs, parallelize, and diagnose flakiness at its root.

Step one: measure

Never guess where the time goes. --durations=N lists the N slowest tests (use --durations=0 for all), and --durations-min=1.0 hides anything faster than a second.

Python Finding the slow tests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The report separates the time spent in the test body (call) from setup and teardown, which tells you whether a slow fixture is the real culprit.

Step two: run less

During development you rarely need the whole suite. Select tests by name, marker or last outcome:

Command Runs
pytest tests/unit/test_cart.py one file
pytest tests/unit/test_cart.py::test_add one test (add ::Class::test for methods)
pytest -k "cart and not slow" tests whose name matches an expression
pytest -m "not integration" tests by marker
pytest --lf only tests that failed last time (--last-failed)
pytest --ff failures first, then the rest
pytest -x / --maxfail=3 stop at the first (or third) failure
pytest --sw stepwise: stop at a failure, resume from it next time
pytest --co -q only list what would be collected
Python Selecting tests with -k and -m

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Expressions combine with and, or, not and parentheses. -k matches test names (and class and module names), -m matches markers, so markers are the more reliable choice for categories you care about.

Step three: pay for setup once

Many slow suites are slow because of repeated setup: creating a database, loading a big file, starting a server for every single test. Fixture scopes let several tests share one instance: function (default), class, module, package and session. The cost is shared state, so use wider scopes only for things tests do not mutate, or reset what they change.

Python Counting how often setup runs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Five tests built five tables at function scope and one at module scope. With a database or a browser session instead of a dictionary, the saving is dramatic.

Other common speed-ups:

  • Avoid real I/O and sleeping. Replace network, clock and sleep with fakes; use in-memory databases.
  • Do less per test. Build the smallest object that proves the point; avoid loading whole fixtures for one assertion.
  • Run in parallel with pytest-xdist (-n auto), once tests are properly isolated (the plugin tour covers it).
  • Split the pipeline: fast unit tests on every push, slow end-to-end tests nightly or on merge (CI lesson).
  • Profile Python imports with python -X importtime when startup, not tests, is slow.

Flaky tests: causes and cures

A flaky test fails sometimes without any code change. Retrying hides the symptom; the cause is almost always one of a few families.

flowchart TB
    F["Flaky test"] --> O["Order / shared state"]
    F --> T["Time and timing"]
    F --> R["Randomness"]
    F --> E["External services"]
    F --> C["Concurrency"]
    O --> O1["fresh fixtures, no module globals"]
    T --> T1["inject / freeze the clock, no real sleeps"]
    R --> R1["seed or inject the random source"]
    E --> E1["fakes, local test doubles"]
    C --> C1["events instead of sleeps, join threads"]

Each family has a standard cure; retries treat none of them.

Python A random test made deterministic

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

We deselected the flaky version with -k "not flaky" because it would sometimes pass and sometimes fail: run the whole file a few times (replace the -k argument) to see it misbehave.

A diagnosis checklist

  1. Reproduce. Run the single test many times: pytest tests/test_x.py::test_y --count=50 (with pytest-repeat), or in a shell loop. Run it with random order and the suite's seed (-p randomly --randomly-seed=...).
  2. Run it alone and in the full suite. Passes alone, fails in the suite means shared state or order; fails only on CI means environment differences (timezone, locale, CPU speed, missing file, port clash).
  3. Look for time: sleep, datetime.now(), timeouts that are too tight on a slow runner.
  4. Look for hidden inputs: random, set/dict ordering you assumed, environment variables, files in the working directory, the current user.
  5. Look for concurrency: threads, subprocesses, async tasks racing each other; replace sleeping with explicit synchronization.
  6. Quarantine while you fix: mark with @pytest.mark.flaky(reruns=2) or xfail(strict=False, reason="issue #123"), with a ticket, a deadline and a visible count. Quarantine is a loan, not a gift.

Other reliable habits: use tmp_path instead of fixed file names, bind servers to port 0 and read the assigned port, compare floats with pytest.approx, sort before comparing unordered results, and keep timeouts generous in CI but add pytest-timeout so nothing hangs.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does pytest --durations=5 show?

    --durations lists the slowest items, split by phase, so you can see whether fixtures or test bodies cost the time.
  2. 2. How do you run only tests that failed in the previous run?

    --lf (--last-failed) uses the cache from the last run to select the failures. --ff runs them first.
  3. 3. A module-scoped fixture builds an expensive object. What is the main risk?

    Wider scopes save setup time but share state, so such fixtures should be read-only or reset what tests change.
  4. 4. A test passes alone but fails in the full suite. What is the most likely cause?

    That symptom usually means another test leaves state behind. Random ordering and fresh fixtures expose and fix it.

Summary

  • Measure with --durations, then optimize what is actually slow.
  • Run less during development: -k, -m, --lf, --ff, -x, --sw, and run all in CI.
  • Share expensive read-only setup with wider fixture scopes; remove real I/O, sleeps and network.
  • Flaky tests come from shared state, time, randomness, external services and concurrency; each has a cure.
  • Quarantine flaky tests temporarily with a ticket, and never normalize "re-run until green".

Next: Test Design and Best Practices