Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 40 ยท Testing and Tooling

Testing with Test.jl

Write tests in Julia with the Test standard library: @test, @testset, @test_throws, approximate equality, test files for packages, and continuous integration.

A test is a small program that checks another program. You write down what the code should do, run the checks, and learn immediately when a change breaks something. That is worth more in Julia than in many languages, because the same code is often reused by other people's packages through multiple dispatch, and a quiet change in one function can ripple a long way.

Julia ships a testing framework in the standard library, called Test. It is small: a handful of macros that you can learn in an afternoon. In this lesson you will write assertions with @test, group them in test sets, check for errors with @test_throws, compare floating-point numbers properly, organize tests for a package, and run them in continuous integration. Julia is not installed on this site, so the Julia code is read-only. The runnable examples are Python stand-ins that rebuild the core ideas of @test and @testset so you can see what the macros do.

@test

Load the library with using Test. Then @test takes any expression that gives true or false:

JULIA
using Test

@test 1 + 1 == 2
@test "julia"[1] == 'j'
@test sqrt(16) == 4
@test [1, 2, 3] == [1, 2, 3]

If the expression is true, the test passes quietly (in the REPL it returns a Test.Pass object). If it is false, you get a report that shows what was actually computed, which is the most useful part:

JULIA
@test 1 + 1 == 3
# Test Failed at none:1
#   Expression: 1 + 1 == 3
#    Evaluated: 2 == 3

Because @test is a macro, it looks inside the expression to build that "Evaluated" line. This is why you should write @test f(x) == 5 rather than computing result = (f(x) == 5) first and testing the boolean. A failure in the REPL throws an error; inside a test set it is recorded and the run continues.

@testset: grouping and a summary

A test set groups related tests under a name and prints a table at the end:

JULIA
@testset "arithmetic" begin
    @test 2 * 3 == 6
    @test 7 รท 2 == 3
    @test 7 % 2 == 1
    @test 2^10 == 1024
end

The output looks like this (the time varies):

Text
Test Summary: | Pass  Total  Time
arithmetic    |    4      4  0.0s

If a test fails, the set does not stop at the first failure. It runs every test and then reports how many passed, failed and errored, so one run shows you all the problems. Test sets can be nested, and the summary then shows each level, which keeps big suites readable. You can also build tests in a loop, and each iteration gets its own result:

JULIA
@testset "squares" begin
    for n in 1:5
        @test n^2 == n * n
    end
end

The core of Test.jl

@test Check that an expression is true.
@testset Group tests and print a summary.
@test_throws Check that the right error occurs.
isapprox / โ‰ˆ Compare floats safely.
1 / 2

Testing for errors

Sometimes the correct behavior is an error. @test_throws takes the expected exception type and an expression:

JULIA
@testset "errors" begin
    @test_throws DomainError sqrt(-1)
    @test_throws BoundsError [1, 2, 3][5]
    @test_throws MethodError length(5, 6)
    @test_throws ArgumentError parse(Int, "abc") # parse throws ArgumentError for bad input
end

The test passes only if the expression throws an exception of that type. If no error occurs, or a different one does, the test fails. You can pass a specific exception value or a predicate when the type is not enough, and for your own exceptions (see Errors and Exceptions) the type is usually all you need. Related macros check output: @test_logs verifies messages sent to the logger, and @test_warn checks text written to stderr.

Comparing floating-point numbers

Computers store most decimals approximately, so exact equality is a trap:

JULIA
0.1 + 0.2 == 0.3          # false, since 0.1 + 0.2 is 0.30000000000000004

Use isapprox, or its operator form โ‰ˆ, which you type as \approx followed by Tab:

JULIA
@test 0.1 + 0.2 โ‰ˆ 0.3
@test isapprox(0.1 + 0.2, 0.3)
@test sin(ฯ€) โ‰ˆ 0 atol = 1e-12

By default isapprox uses a relative tolerance of about 1.5e-8, which suits numbers of normal size. It breaks down when you compare something with exactly zero: relative to zero, any difference is huge. That is the reason for the last line above. Give an absolute tolerance with atol, and rtol sets a different relative one. It works on arrays element by element, and @test A โ‰ˆ B is the standard way to compare matrices.

Organizing tests in a package

A package keeps its tests in a test folder, and the entry point is test/runtests.jl:

Text
MyPackage/
โ”œโ”€โ”€ Project.toml
โ”œโ”€โ”€ src/
โ”‚   โ””โ”€โ”€ MyPackage.jl
โ””โ”€โ”€ test/
    โ””โ”€โ”€ runtests.jl
JULIA
# test/runtests.jl
using MyPackage
using Test

@testset "MyPackage" begin
    include("test_geometry.jl")
    include("test_parsing.jl")
end

Run them from the package REPL with ] test, or from code with using Pkg; Pkg.test(). This creates a clean temporary environment that contains your package and the test dependencies, so you find missing dependencies before your users do. Test-only packages go in test/Project.toml (or the older [extras] and [targets] sections of the main Project.toml), so they are not forced on everyone who installs your package. See Packages and Environments for how environments work.

A few habits make suites better:

  • Test the behavior you promise, not how the code is written.
  • Give each test set a name that says what is being checked.
  • Test edge cases: empty input, one element, negative numbers, the boundary values.
  • Seed random numbers so a failure can be reproduced (see Random Numbers and Statistics).
  • When you fix a bug, add a test that fails without the fix.

@test_broken records a bug you know about. It "passes" while the code is still wrong and starts to fail when the bug is fixed, which tells you to turn it into a normal @test.

Continuous integration

Tests are most useful when they run automatically on every change. Julia has ready-made actions for GitHub. This workflow file, saved as .github/workflows/CI.yml, runs the suite on several Julia versions and operating systems:

YAML
name: CI
on: [push, pull_request]

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
        julia-version: ['1.10', '1']
    steps:
      - uses: actions/checkout@v4
      - uses: julia-actions/setup-julia@v2
        with:
          version: ${{ matrix.julia-version }}
      - uses: julia-actions/cache@v2
      - uses: julia-actions/julia-runtest@v1

The '1' entry means "the latest Julia 1.x release". Testing on the oldest version you support, as well as the newest, catches compatibility mistakes. Add the julia-processcoverage and codecov steps if you want to see which lines the tests execute.

A tiny test framework in Python

The best way to understand a macro-based framework is to build a miniature one. Python has no macros, so the stand-in below takes the expression as a lambda and reports it, but the logic matches: record every result, keep going after a failure, and print a summary table at the end.

Python A mini @testset and @test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that the run continues after the first failure and reports all of them together, as @testset does. The second stand-in shows why โ‰ˆ exists. It uses Python's math.isclose with the same default relative tolerance as Julia's isapprox, which is the square root of machine epsilon.

Python Why isapprox needs atol near zero

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The relative tolerance alone rejects a number that is about 1e-16 away from zero, because relative to zero every difference is infinitely large. Adding an absolute tolerance fixes it, in Python with abs_tol and in Julia with atol.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does @testset "name" begin ... end add on top of plain @test calls?

    A test set records every result and prints a table of passes, failures and errors at the end, instead of stopping at the first failure.
  2. 2. Which test passes only when the code throws a DomainError?

    @test_throws takes the exception type and the expression, and it fails if the error does not happen.
  3. 3. Why is @test sin(ฯ€) โ‰ˆ 0 likely to fail without an extra argument?

    Use an absolute tolerance, for example @test sin(ฯ€) โ‰ˆ 0 atol = 1e-12, when comparing with zero.
  4. 4. Where does Julia look for a package's tests when you run ] test?

    The entry point is test/runtests.jl, which usually includes other test files.
  5. 5. What is @test_broken for?

    It marks a test that is expected to fail, so the suite stays green now and reminds you to promote it once the code works.

Summary

  • using Test gives you @test for assertions; a failure shows the evaluated expression.
  • @testset groups tests, continues after failures and prints a pass/fail summary.
  • @test_throws ExceptionType expr checks that the right error occurs.
  • Compare floats with isapprox or โ‰ˆ, and add atol when comparing against zero.
  • Package tests live in test/runtests.jl and run with ] test; CI actions run them on several Julia versions and systems.

Next: find and fix problems with Debugging and Profiling.