Skip to content
Tutorials Center
My learning

Topics

Lesson 101 of 147 Β· Testing and Tooling

Benchmarks and Fuzzing

Measure Go code with testing.B and b.Loop, read ns/op and allocs/op, compare runs with benchstat, find bugs with native fuzzing and keep documented examples verified as tests.

Unit tests answer "is the code correct?". Two other questions come up often enough that Go's testing package answers them too. Benchmarks answer "how fast is it, and how much memory does it allocate?" with numbers you can compare before and after a change. Fuzzing answers "is there an input I never thought of that breaks it?" by letting the machine invent thousands of inputs per second. A third feature, examples, keeps the code samples in your documentation honest by running them as tests.

All three live in _test.go files next to your code and run with the same go test command you met in the testing lesson and table-driven tests. This site cannot run go test, so each section shows the real test file as a read-only snippet and then runs the same ideas inside main using the testing package's public functions. The program is a stand-in for the file; the code that matters is identical.

Timings depend on your machine, so this lesson never quotes a speed from the runnable examples. They print things that are the same on every run: allocation counts, custom metrics and pass/fail results. In your own projects, the nanoseconds are the point, and you should always compare numbers measured on the same machine in the same conditions.

Your first benchmark

A benchmark is a function named BenchmarkXxx that takes a *testing.B. Since Go 1.24 the recommended body is a for b.Loop() loop:

Go
func BenchmarkConcatPlus(b *testing.B) {
	parts := makeParts(50)
	for b.Loop() {
		concatPlus(parts)
	}
}

You run it with go test -bench=. -benchmem. The framework calls your function several times with a growing iteration budget until the measurement is stable (about one second by default), then divides the total time by the number of iterations. Typical output looks like this (the numbers are an example; yours will differ):

TEXT
goos: linux
goarch: amd64
pkg: example.com/strs
cpu: Example CPU @ 3.00GHz
BenchmarkConcatPlus-8       200000       5400 ns/op     26000 B/op      50 allocs/op
BenchmarkConcatBuilder-8   3000000        390 ns/op      1400 B/op       6 allocs/op
PASS

Reading a line from left to right:

  • BenchmarkConcatPlus-8: the benchmark name, and the value of GOMAXPROCS (8 here).
  • 200000: how many iterations the final measurement ran.
  • 5400 ns/op: average nanoseconds per iteration. Lower is faster.
  • 26000 B/op: average bytes allocated on the heap per iteration (only with -benchmem or b.ReportAllocs()).
  • 50 allocs/op: average number of separate heap allocations per iteration. This is often the number that explains the time: allocations are what make the garbage collector work (see garbage collection and the stack and the heap).

Why b.Loop and not b.N

Before Go 1.24 you wrote for i := 0; i < b.N; i++ { ... }, and the framework chose b.N for you. That style has traps that b.Loop removes:

  • Setup is free. With b.Loop, everything before the loop runs once and is excluded from the timing, because the timer starts on the first call of b.Loop() and stops when it returns false. With b.N you had to call b.ResetTimer() yourself, and forgetting it skewed the numbers.
  • The compiler can't delete your work. If the result of a call is never used, an optimizing compiler may inline the call and discard it, and you then benchmark an empty loop. b.Loop keeps the calls inside the loop alive. Older code assigned the result to a package-level sink variable for the same reason.
  • The function runs once per measurement. The benchmark function is called a single time per run with b.Loop, so setup code isn't repeated for every trial.

Both styles still work, and you will meet b.N loops in older projects.

Running a benchmark from main

The function testing.Benchmark runs a benchmark function programmatically and returns a testing.BenchmarkResult with the fields N, T, MemAllocs and MemBytes and the methods NsPerOp(), AllocsPerOp() and AllocedBytesPerOp(). The program below compares two ways to join 51 strings. It prints the allocation counts, which don't change between runs, and a sanity check that something was measured:

Go testing.Benchmark and allocs/op

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The run prints 49, 7 and 1 allocations per operation. The += version allocates a brand-new string on every step because strings are immutable (strings and runes). strings.Builder appends to a growing byte buffer, so it allocates only when the buffer is full. Telling the builder the final size with Grow removes even those reallocations, leaving the single allocation that String() needs to be handed out without copying. This is the usual story with Go performance: fewer allocations first, micro-tuning later.

Measuring allocations without a benchmark

If all you want to know is "how many allocations does this call make?", testing.AllocsPerRun is simpler than a benchmark. It runs the function once to warm up, then runs it a given number of times and returns the average count of allocations as a float64. Unlike a timing it is exact, so it is also a good way to put a regression test around a hot path:

Go
func TestNoAllocs(t *testing.T) {
	allocs := testing.AllocsPerRun(100, func() { Lookup("key") })
	if allocs != 0 {
		t.Errorf("Lookup allocates %v times per call, want 0", allocs)
	}
}

The next program counts allocations for converting a number to a string. The surprise is in the small numbers: the standard library keeps ready-made strings for 0 to 99, so converting 42 costs nothing at all:

Go testing.AllocsPerRun

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Concatenating "id-" + Itoa(...) costs two: one for the digits, one for the joined string. fmt also allocates, because it has to box the int in an any parameter, so strconv is the lighter choice in hot loops. In most code the difference doesn't matter. Measure first, as the next section's rules explain.

Custom metrics, timers and sub-benchmarks

Sometimes "per operation" isn't the number you care about. b.ReportMetric(value, "unit/op") adds your own column to the output, for instance the number of comparisons a sort made, or the hit rate of a cache. The program below benchmarks a sort and reports a custom metric. Comparison counts are the same on every run, so the output is stable:

Go b.ReportMetric and excluding setup

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

A few more tools from *testing.B:

  • b.ReportAllocs() turns on the B/op and allocs/op columns for one benchmark without needing -benchmem.
  • b.StopTimer() and b.StartTimer() pause the clock around work that shouldn't count. Each call has a cost of its own, so don't use them inside a tight loop. Prefer doing the setup once before the loop.
  • b.SetBytes(n) says each operation processes n bytes, and adds an MB/s column. Good for parsers and encoders.
  • b.Run(name, f) creates sub-benchmarks, the benchmark twin of subtests. This is how you measure how a function scales with its input:
Go
func BenchmarkSort(b *testing.B) {
	for _, size := range []int{10, 1_000, 100_000} {
		b.Run(fmt.Sprintf("n=%d", size), func(b *testing.B) {
			data := randomInts(size) // setup, runs once and is not timed
			work := make([]int, size)
			for b.Loop() {
				copy(work, data)
				slices.Sort(work)
			}
		})
	}
}

Each sub-benchmark appears as BenchmarkSort/n=10, BenchmarkSort/n=1000 and so on, and you can pick one with -bench 'Sort/n=1000'.

Benchmarking honestly

Microbenchmarks are easy to run and easy to get wrong. A few rules keep you out of trouble:

  • Measure before you optimize. Use a profile (profiling and debugging) to find the slow part of the whole program. A function that takes 3% of the run time can't give you more than 3%, however well you tune it.
  • Run each variant several times. go test -bench=. -count=10 repeats every benchmark ten times so you can see the spread. One run is an anecdote.
  • Compare on a quiet machine. Close the browser, plug in the laptop, and don't change anything between the "before" and "after" runs.
  • Benchmark realistic inputs. A sort benchmark with 10 items says little about a sort of a million. A cache benchmark with a 100% hit rate is not a cache benchmark.
  • Use -benchtime to change how long each benchmark runs: -benchtime=3s for more precision, or -benchtime=1000x for exactly 1000 iterations.
  • Keep the result alive. With b.Loop this is automatic. In an old-style b.N loop, assign the result to a package-level variable.

Comparing runs with benchstat

When you change code, save the numbers before and after, then let the benchstat tool decide whether the difference is real or just noise. It isn't part of the standard distribution, but it is maintained by the Go team:

Terminal
go install golang.org/x/perf/cmd/benchstat@latest

go test -bench=. -benchmem -count=10 > old.txt
# ... change the code ...
go test -bench=. -benchmem -count=10 > new.txt

benchstat old.txt new.txt

It prints a table with the median of each metric, the confidence interval, and a ~ where the change is not statistically significant. The output has the shape below (again, example numbers):

TEXT
                β”‚   old.txt    β”‚               new.txt               β”‚
                β”‚    sec/op    β”‚   sec/op     vs base                β”‚
ConcatPlus-8      5.40Β΅ Β± 1%     0.39Β΅ Β± 2%   -92.78% (p=0.000 n=10)

A change with p above 0.05, or one reported as ~, is indistinguishable from noise. Don't ship a "speedup" that benchstat can't see.

Fuzzing: let the computer find the bad input

Table tests only check the inputs you thought of. A fuzz test checks a property that must hold for every input, and the Go tool generates inputs for you. It starts with the seeds you provide, mutates them, and uses coverage feedback to steer towards inputs that reach new code. A fuzz target is named FuzzXxx and takes a *testing.F:

Go
func FuzzReverse(f *testing.F) {
	// Seed corpus: these always run, even in a plain `go test`.
	f.Add("hello")
	f.Add("hΓ©llo, δΈ–η•Œ")
	f.Add("")

	f.Fuzz(func(t *testing.T, s string) {
		rev := Reverse(s)
		if utf8.ValidString(s) && !utf8.ValidString(rev) {
			t.Errorf("Reverse(%q) = %q is not valid UTF-8", s, rev)
		}
		if again := Reverse(rev); again != s {
			t.Errorf("Reverse(Reverse(%q)) = %q", s, again)
		}
	})
}

The inner function is called with each input, and its parameters (after t) must be of the supported types: string, []byte, bool, byte, rune, all the integer and float types. You can't assert an exact output, because you don't know the input in advance. Instead you check properties:

  • Round trip: Decode(Encode(x)) == x.
  • Idempotence: formatting already-formatted code changes nothing.
  • Invariants: the output of a sort is sorted and has the same length; a parser never panics.
  • Two implementations agree: a fast version and an obviously correct slow one.

You run it with a time limit, because fuzzing is open-ended:

Terminal
go test                             # runs only the seed corpus (and saved failures)
go test -fuzz=FuzzReverse -fuzztime=30s

When the fuzzer finds a failing input, it minimizes it, saves it as a file in testdata/fuzz/FuzzReverse/, and prints how to run it again:

TEXT
--- FAIL: FuzzReverse (0.02s)
    --- FAIL: FuzzReverse (0.00s)
        reverse_test.go:14: Reverse("Γ©") = "\xa9\xc3" is not valid UTF-8

    Failing input written to testdata/fuzz/FuzzReverse/582528ddfad69eb5
    To re-run:
    go test -run=FuzzReverse/582528ddfad69eb5
FAIL

Commit that file. From then on, a plain go test replays it, so the bug can never come back unnoticed. The file is simple text:

TEXT
go test fuzz v1
string("Γ©")

The idea, runnable

The real fuzzing engine needs the go test command, so the program below uses the same idea with a hand-written generator. It builds random strings from a small alphabet with a fixed seed, so the run is repeatable, and checks the two properties against a buggy Reverse that reverses bytes instead of runes:

Go Finding a bug with random inputs

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

A real fuzzer does the same thing far more cleverly: it remembers which inputs reached new branches, mutates them, shrinks failures to the smallest example, and runs on all your cores. Tips for good fuzz targets:

  • Keep the target fast and deterministic. No sleeping, no network, no global state that changes between calls. Thousands of runs per second is the goal.
  • Fuzz parsers and decoders first. Anything that takes untrusted bytes (JSON, images, protocols, your own file format) is where fuzzing finds crashes and hangs.
  • Treat a panic as a failure. You don't need to check for it; an uncaught panic fails the test and saves the input.
  • Seed with real examples. The better the seed corpus, the faster the fuzzer reaches interesting code.

Fuzzing can run a very long time and fill a disk with a cache. The tool stores generated inputs in the Go build cache (go env GOCACHE), not in your repository. Only the failing inputs, which you decide to keep, go into testdata/fuzz.

Examples that run as tests

A function named ExampleXxx is both documentation and a test. If its last comment starts with // Output:, go test runs the function, captures what it prints, and fails if the text differs. The same function appears under the identifier on pkg.go.dev, so the sample code on your documentation page is always correct (documentation and examples covers naming rules):

Go
func ExampleSlug() {
	fmt.Println(Slug("Hello, World!"))
	fmt.Println(Slug("  Go  "))
	// Output:
	// hello-world
	// go
}

Use // Unordered output: when the lines can appear in any order, for example when you print a map's contents. The testing package can run examples from main as well, so here is the real mechanism at work. The second example has a wrong expected output, so one passes and one fails:

Go Examples as tests (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

In a real project you never write the InternalExample list: go test finds the Example functions and their // Output: comments for you. The failure message prints got: and want: blocks, so you can see the difference. Examples are the cheapest tests you will ever write, and they double as the first thing a new user reads.

Where each kind of test belongs

Question Tool Command
Does it work for these inputs? TestXxx with a table go test
Does it work for any input? FuzzXxx go test -fuzz=FuzzXxx
How fast is it? How many allocations? BenchmarkXxx go test -bench=. -benchmem
Is the documentation still true? ExampleXxx go test

Check your understanding

Quick check Β· 5 questions

of 5 answered You scored / 5 Β· perfect!

  1. 1. What does the allocs/op column of a benchmark report?

    allocs/op is the number of separate heap allocations per operation; B/op is the number of bytes. Both are averages over all iterations.
  2. 2. What is a benefit of for b.Loop() over for i := 0; i < b.N; i++?

    b.Loop (Go 1.24) starts the timer on its first call, keeps the loop body from being optimized away and runs the benchmark function once per measurement.
  3. 3. Which of these is a good property for a fuzz test of an Encode/Decode pair?

    A fuzz test can't know the exact output for generated input, so it checks properties that must always hold, such as a round trip.
  4. 4. A fuzzer finds a crashing input. What should you do with the file it wrote to testdata/fuzz/?

    Saved failing inputs become part of the seed corpus. Committing them turns every found bug into a permanent regression test.
  5. 5. How does go test decide whether an ExampleXxx function passes?

    The function's standard output is captured and compared, ignoring leading and trailing space, with the // Output: comment. Without that comment the example is compiled but not run.

Summary

  • A benchmark is func BenchmarkXxx(b *testing.B); write the body as for b.Loop() { ... } (Go 1.24+) and run it with go test -bench=. -benchmem.
  • ns/op is time per operation, B/op is bytes allocated and allocs/op is the number of allocations; fewer allocations usually explains a speedup.
  • testing.Benchmark and testing.AllocsPerRun work in ordinary programs; AllocsPerRun is exact and makes a good regression test.
  • Use b.ReportMetric for custom numbers, b.Run for sizes, -count=10 for repetitions and benchstat to tell a real change from noise.
  • A fuzz test (FuzzXxx, f.Add, f.Fuzz) checks properties for generated inputs; failures are saved in testdata/fuzz and should be committed.
  • ExampleXxx functions with // Output: are documentation and tests in one.

Next: Profiling and Debugging