Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 38 ยท Getting Started

Running Tests

Master the pytest command line: run one file, one test or a keyword match, stop at the first failure, control output and tracebacks, and read exit codes.

Running every test after every edit is fine for a small project, but a real suite can hold thousands of tests. You need to run just the test you are working on, stop at the first failure, see more or less output, and understand what the process tells scripts when it exits. pytest has a compact set of command-line options for all of this.

In the examples, the extra arguments after the file dictionary are exactly what you would type after pytest in a terminal.

Which tests to run

You can point pytest at a directory, a file, or one exact test. The exact name is called a node id: the file path, then ::, then the function (and class, if any).

Terminal
pytest                       # everything under the current folder
pytest tests/                # one directory
pytest tests/test_cart.py    # one file
pytest tests/test_cart.py::test_total      # one test
pytest tests/test_cart.py::TestCart::test_add   # one method in a class

The next example has a small suite. We run only one node id.

Python Run a single test by node id

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Only one test ran. Replace the node id with test_basic.py::test_addition, or with just test_basic.py::TestStrings to run both methods of the class.

Selecting by keyword: -k

-k matches a substring (or a small expression) against test names, class names and file names. It is the fastest way to say "run the tests about X".

Python -k selects by name

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

-k "login" runs two tests. The expression user and not logout supports and, or, not and parentheses, so you can express precise selections. pytest also reports how many tests were deselected, so you can see that nothing was lost.

Stop early

When something is badly broken, a hundred red tests are noise. Two options help:

  • -x (or --exitfirst): stop after the first failure.
  • --maxfail=3: stop after three failures.
Python Stop at the first failure

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The report says stopping after 1 failures, and test_three and test_four never ran. Use -x when you fix problems one at a time.

How much to show

Option Effect
-q quiet: dots and a short summary
-v verbose: one line per test with its node id
-vv more detail, for example full diffs without truncation
-rA summary of every test with its outcome
-rf summary of failed tests only
--tb=short shorter tracebacks
--tb=short one line per failure
--tb=no no tracebacks at all
-s do not capture output, so print shows immediately

By default pytest captures what your tests print and shows it only for failing tests. That keeps passing runs clean. Use -s when you are debugging with print, or -rP to show captured output for passing tests too.

Python Captured output and -rP

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In the first run the passing test shows nothing. The second one adds a "PASSES" section with the captured text of the passing test, and the failing test always shows its output under "Captured stdout call".

Look without running

--collect-only lists what pytest would run. Combined with -q it prints one node id per line, which is great for checking that your naming and selection work.

Python --collect-only

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Rerunning failures

On your machine pytest remembers the last run in a .pytest_cache folder. Two options use it:

  • --lf (--last-failed): run only the tests that failed last time.
  • --ff (--failed-first): run the failed ones first, then the rest.
  • --sw (--stepwise): stop at a failure and continue from it next time.

These need the cache plugin, which the in-browser helper turns off, so they appear here as read-only commands. Try them on your own project; they turn a slow suite into a fast fix loop.

Exit codes

When pytest finishes it returns an exit code, which scripts and CI use:

Code Meaning
0 all tests passed
1 some tests failed
2 the run was interrupted by the user
3 an internal error
4 the command line was used incorrectly
5 no tests were collected

Exit code 5 surprises people: an empty selection is not a pass. The helper prints the code, so you can see it for yourself by using -k nothing_matches.

Putting options in a config file

Typing -v --tb=short every time gets old. Put your habits in pytest.ini, pyproject.toml or tox.ini:

INI
# pytest.ini
[pytest]
addopts = -ra --tb=short

addopts is added to every command line. You will learn more about configuration in the next lesson.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does pytest tests/test_cart.py::test_total run?

    A node id after :: selects one test exactly. Use -k for a name search across files.
  2. 2. Which option stops the run after the first failing test?

    -x (--exitfirst) stops at the first failure. --maxfail=N stops after N failures.
  3. 3. Why does a passing test not show its print output by default?

    Output capture keeps clean runs clean. Use -s to see output live or -rP to include it for passing tests.
  4. 4. What is the exit code when pytest collects no tests?

    Exit code 5 means that no tests were collected. Treating an empty run as success could hide a broken test setup.
  5. 5. What does -k "user and not logout" do?

    -k accepts a boolean expression of name fragments.

Summary

  • Select tests by directory, file, node id (file::class::test) or keyword (-k).
  • -x and --maxfail stop early; --lf and --ff reuse the last run's failures.
  • -q, -v, --tb=... and -r... control how much you see; -s disables capturing.
  • --collect-only shows what would run without running it.
  • Exit code 0 is success, 1 failures, 5 nothing collected.
  • Put repeated options in addopts in a config file.

Next: Test Discovery and Project Layout