Skip to content
Tutorials Center
My learning

Topics

Lesson 70 of 95 ยท Projects

Project: A Mini Test Framework

Build a test framework in Bash: assert functions, automatic test discovery with declare -F, subshell isolation, temp-dir fixtures, skips and exit codes.

Scripts that delete files, rename things or talk to servers deserve tests, and in the Bash world the usual answer is a framework such as bats or shunit2. Those tools are not magic. In this project you write your own in about 100 lines, and in doing so you learn the machinery they all share: assertion functions, discovering tests by looking at the functions the shell knows about, isolating every test in a subshell and a fresh directory, and turning the results into a report and an exit code that a build system can act on.

You will use functions, subshells, exit codes and set -e, traps, parameter expansion and arrays. The framework is also a good excuse to see how set -e really behaves, because test frameworks live and die by it.

The plan

TEXT
  lib_test.sh                                   minitest.sh
  -----------                                   -----------
  source ./minitest.sh   ------------------->   assert_eq, assert_contains, run, skip
  test_a() { ... }
  test_b() { ... }       <--- declare -F ----   mt_discover   finds every test_* function
  setup() / teardown()   <--------------------  mt_run_one    one subshell + one temp dir each
  mt_main "$@"           ------------------->   mt_main       report, summary, exit code

The rules of the game, which are the same as in most test frameworks:

  1. A test is a function whose name starts with test_.
  2. A test passes if it finishes with status 0 and fails otherwise. Any failing command fails the test, which is what set -e is for.
  3. Assertions are helpers that end the test with a clear message when a comparison is false.
  4. Every test runs in its own subshell and its own empty directory, so tests cannot affect each other.
  5. The runner prints one line per test, a summary, and exits non-zero if anything failed.

Step 1: an assertion is a command that can fail

At bottom a test is a command with an exit status. An assertion is a function that returns quietly when the claim is true and prints an explanation and stops when it is false. Stopping means exit, and exit ends the whole shell, which is why each test must run inside parentheses: the ( ... ) subshell takes the exit instead of your runner.

Bash Step 1: assert_eq and a loop that runs tests in subshells

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

${1@Q} is a Bash 5 expansion that quotes the value so it can be pasted back into a shell. In a failure message it makes whitespace and special characters visible: 'a b' is easy to tell apart from 'a b'.

Run the failing test without the parentheses and exit 1 would end the whole script before the loop could report anything. That is the first reason for isolation.

Step 2: discovery, declare -F and the set -e trap

How does the runner find the tests? The shell already keeps a table of every function it knows. declare -F prints just the names, one per line in the form declare -f NAME, sorted alphabetically. So discovery is "read that list and keep the names that start with test_". compgen -A function test_ does the same in one command and is handy interactively.

The second question is how a test fails when a command in the middle fails. Bash's answer is set -e, but it has a rule that surprises people: set -e is switched off inside any command whose status is being tested: the condition of if, the left side of && or ||, and everything those commands call. The next example shows the difference with the same function, and also that a subshell protects the runner's variables and working directory.

Bash Step 2: declare -F, set -e inside if, and subshell isolation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

This is the reason the framework below runs each test as an ordinary statement, ( set -e; "$name" ) > "$out" 2>&1, and reads $? on the next line. If the runner wrote if ( ... ); then the "failing" command in the middle of a test would be silently ignored, and tests would pass when they should fail. Every set -e based framework has to get this right.

Step 3: the framework

Here is the whole of minitest.sh. It is meant to be sourced by a test file, which then defines test_* functions and ends with mt_main "$@".

Bash Step 3: minitest.sh, a library to test and a test file

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The report is sorted by name, because declare -F is. The isolation tests run in order a then b: the a test creates a file called marker, and the b test checks that it does not exist, which proves the fresh directory.

How it works

Assertions. _mt_fail uses the builtin caller. Called from inside an assertion, caller 1 returns the line, function and file of the code that called the assertion, so the message points to the line in the test file. read -r line func file < <(caller 1) splits that into three variables. All assertions end with exit 1, which ends the test's subshell.

run. output=$("$@" 2>&1) && status=0 || status=$? captures the combined output of any command and its exit status in two variables, without letting a non-zero status stop a set -e test. A test can then say run latest_backup backups and assert on $status and $output, including the failure case. ($status is an ordinary variable in Bash; it is only special in zsh.)

skip. A test that cannot run here (needs the network, needs root) calls skip "reason": it prints the reason and exits with status 77, a number the runner treats as "skipped" instead of "failed". The code 77 is borrowed from the automake test convention.

Discovery. mt_discover loops over declare -F. The optional pattern argument lets you run a subset: bash lib_test.sh slugify.

Isolation and fixtures. mt_run_one creates a directory for the test and runs everything in a subshell that sets strict mode with set -euo pipefail, changes into the directory, registers an EXIT trap that calls teardown (so teardown runs even when the test fails), calls setup if it exists, and finally calls the test. Standard output and standard error go to a file; the runner only prints that file when the test fails or is skipped, so passing tests are quiet. set -u makes a misspelled variable in a test a failure instead of an empty string.

Cleanup. MT_ROOT=$(mktemp -d) is the parent of every work directory, and one EXIT trap removes it when the whole run ends. (The trap lives in the runner shell. The EXIT trap in the subshell is a different one, which is why they do not collide.) A framework that sources into someone else's script should be careful about taking over trap ... EXIT; for a tool you control it is fine.

Reporting. mt_main prints ok, skip or FAIL per test, a summary and finally returns the result of (( failed == 0 )). That arithmetic test is true only if no test failed, so the script's exit status is 0 for success and 1 for failure, which is the contract every CI system understands.

Why a subshell and not a new bash process per test? A subshell is a cheap fork that inherits all functions from the test file. A fresh bash would not know your test_* functions unless you exported them. Real frameworks like bats do start new processes, but they pay for it with a preprocessing step.

Step 4: what a failure looks like

A framework is only convincing when you have seen it fail. This test file has seven tests; five of them are wrong in different ways: a false comparison, an unexpected substring, a failing command in the middle of a test (a quiet grep -q that finds nothing, with echo after it never printed), a test that prints something and then fails, and a failing test with a teardown that still runs. The run exits with status 1. Then the runner is used with a name filter and with a filter that matches nothing.

Bash Step 4: failing tests, filters and exit statuses (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Read the failure output top to bottom and check what each tells you:

  • test_wrong_expectation names the line of the assertion and shows both values quoted, with the optional label (greeting).
  • test_failing_command_stops shows only the teardown line: grep -q failed silently, set -e ended the test right there, and never printed never appeared. The framework tells you the test failed but not where. A more advanced version could install trap 'echo "failed at line $LINENO" >&2' ERR in the test subshell to add that. That is a good exercise.
  • test_output_is_shown proves that output from a failing test is kept and shown, indented.
  • Every failure ends with teardown ran, so cleanup happens on all paths.
  • A filter that matches no test exits with status 1 and "no tests found". A typo in a filter must not look like success.

Testing scripts, not only functions

Everything so far tested functions that were sourced into the test file. To test a script, run it with run and assert on the result (a snippet for your own test file; tools/tool.sh stands for any script of yours, and $OLDPWD is the directory the runner was in before it moved into the test's empty directory):

Terminal
test_script_rejects_missing_argument() {
  run bash "$OLDPWD/tools/tool.sh"
  assert_eq 2 "$status"
  assert_contains "$output" "usage:"
}

This works well when the script follows the habits from Designing Maintainable Scripts: configuration through environment variables (TODO_FILE, BACKUP_NOW) so tests can redirect it, a non-zero status for every kind of failure, and errors on standard error. If a script can only be tested by editing it, the script is the problem. The finished backup and todo projects were written with that in mind.

When it makes sense, test behaviour, not text: assert the exit status and the important part of the output instead of the entire output byte for byte, so that an improved message does not break twenty tests.

Where to go from here

  • Timing: record $EPOCHREALTIME before and after each test and print slow tests. (Not deterministic, so not shown here.)
  • assert_status with a pattern for output, assert_lines for multi-line output, assert_dir.
  • Colour with the NO_COLOR/-t 1 logic from the todo project.
  • TAP output (ok 1 - name), the Test Anything Protocol that many CI tools read.
  • Parallel tests with xargs -P or background jobs and wait, once every test uses its own directory (which you now have).
  • Real tools: bats-core is the standard choice for Bash scripts and gives you @test "name" { ... } syntax, setup_file, helpers and TAP output. shellcheck finds bugs without running anything. Neither is available in our sandbox; the code above is what they are built on.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does the framework run every test inside ( ... )?

    exit ends the current shell. In a subshell that is just the test, so the runner survives to report the result. The subshell also stops tests from leaking variables or cd into each other.
  2. 2. What is wrong with if ( set -e; "$name" ); then pass; else fail; fi as the way to run a test?

    Inside the condition of if (and in &&/|| lists) errexit is disabled, even in functions called from there. The framework runs the test as a plain command and reads $? afterwards.
  3. 3. How does mt_discover find the tests?

    The shell already knows all defined functions. declare -F prints their names (sorted), and the loop keeps the ones that follow the naming convention.
  4. 4. What does a test that calls skip "no network" produce?

    skip prints the reason and exits 77. The runner maps that code to the skip line and counts it separately, so skipped tests neither pass nor fail.
  5. 5. Why is teardown registered with an EXIT trap in the test's subshell instead of simply being called after the test?

    When an assertion calls exit or set -e stops the test, code after the test call never runs. An EXIT trap fires on every way out of the subshell.

Summary

  • A test is a function that exits 0; an assertion is a helper that explains and exits when a claim is false.
  • Discover tests with declare -F (or compgen -A function), filtering on a naming convention such as test_.
  • Run each test in a subshell with set -euo pipefail, in a fresh mktemp -d directory, with setup before and teardown in an EXIT trap.
  • Never run tests inside an if condition: set -e is disabled there. Run the subshell as a plain command and read $?.
  • run captures $output and $status, so failure cases can be asserted too; skip uses status 77.
  • Print one line per test, a summary, and return non-zero when anything failed, so CI can use the script directly.

Next: Project: A Deployment Script Skeleton