Skip to content
Tutorials Center
My learning

Topics

Lesson 104 of 132 ยท Testing and Tooling

Benchmarking Rust Code

Measure Rust performance correctly: Instant timing, warm-up and calibration, black_box, medians and outliers, criterion and divan, and the pitfalls that fool benchmarks.

"Which version is faster?" is one of the easiest questions to ask and one of the hardest to answer honestly. A single Instant::now() around a function call tells you almost nothing: the first run is slower because caches are cold, the optimizer may have deleted the work you wanted to measure, another process may have stolen the CPU for a millisecond, and a debug build can be a hundred times slower than release. Good benchmarking is a method, not a stopwatch.

This lesson shows that method. You will write a small benchmark harness from the standard library (Instant, Duration, black_box), learn why it needs warm-up, calibration and many samples, compute robust statistics (median, spread, outliers), and then see how criterion and divan do the same work for you in a real project. It complements Performance and Optimization, which is about making code fast; this lesson is about proving it.

Our in-browser editor runs on a shared server, in a build with debug assertions, so absolute times from it would be meaningless. In this lesson everything printed is deterministic: counts, checksums and statistics computed from fixed numbers. Run the harnesses on your own machine with cargo run --release to see real timings.

What a benchmark has to do

flowchart LR
    A["Warm-up<br/>fill caches, train predictors"] --> B["Calibrate<br/>pick iterations per sample"]
    B --> C["Collect samples<br/>time many batches"]
    C --> D["Statistics<br/>median, spread, outliers"]
    D --> E["Compare against a baseline"]

A trustworthy benchmark is a pipeline, not one measurement.

Each stage answers a specific problem:

  • Warm-up removes first-run effects: page faults, cold CPU caches, lazy initialization, CPU frequency ramp-up.
  • Calibration chooses a batch size so one sample lasts long enough (about a millisecond or more) that the clock's resolution and call overhead stop mattering.
  • Many samples let you see variation instead of trusting one lucky or unlucky run.
  • Robust statistics (median rather than mean) keep a few outliers from dominating.
  • A baseline turns a number into a conclusion: faster or slower than what?

Instant and Duration

std::time::Instant is a monotonic clock: it never goes backward, so it is the right tool for measuring elapsed time (unlike SystemTime, which can jump when the clock is adjusted). Instant::now() takes a reading and elapsed() gives a Duration. A Duration supports arithmetic and conversion, which is all a harness needs. See Time and Dates for the basics.

Rust Duration arithmetic for per-iteration cost

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

Notice the prints: we show the arithmetic on a fixed Duration but only a boolean about the real elapsed time, because the real number would change on every run.

Statistics that survive noise

The mean of timing samples is easily ruined by one slow sample (the OS scheduled something else). The median ignores outliers, and the interquartile range (IQR) measures typical spread. A common rule flags a sample as an outlier if it is more than 1.5 IQRs beyond the quartiles. Here is the whole toolbox, applied to a fixed set of "timings" in nanoseconds with one bad sample:

Rust Median, spread and outliers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

The mean is about 28 ns off the typical value because of a single bad sample, while the median hardly moves. Always report the median (or a low percentile) plus a measure of spread, and treat a wide spread as a sign the measurement is noisy rather than as a result.

black_box: keep the optimizer honest

The optimizer is allowed to delete code whose result is unused and to compute things at compile time when the inputs are constants. A benchmark that calls fib(20) with a literal and ignores the result may measure nothing at all. std::hint::black_box is an identity function that the compiler must treat as opaque: pass your input through it so it cannot be folded into a constant, and pass your result through it so it cannot be thrown away.

Now we can assemble a harness. It warms up, calibrates the batch size until one batch takes at least a target duration, collects samples and summarizes them.

Rust A small benchmark harness

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

This is, in miniature, what criterion does: warm-up, an automatically chosen iteration count, many samples, robust statistics and outlier classification. On your own machine you would print report.median_per_iteration() too.

Comparing algorithms by counting work

Time is noisy, but work is exact. When you are deciding between two algorithms, counting comparisons, allocations or bytes moved gives a stable answer that is the same on every machine, and often explains the timing result later.

Rust Linear versus binary search by comparisons

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

The counts grow like n and log n. Whether the binary search is also faster in seconds for small inputs depends on caches and branch prediction, which is exactly why you still measure; the counts tell you what to expect and what to look for.

criterion in a real project

criterion is the long-standing statistics-driven benchmarking crate. It lives in benches/, runs on stable Rust, saves previous results and tells you whether a change is statistically significant. Read-only setup:

TOML
# Cargo.toml
[dev-dependencies]
criterion = "0.5"

[[bench]]
name = "search"
harness = false        # criterion provides its own main()
Rust
// benches/search.rs
use criterion::{criterion_group, criterion_main, BenchmarkId, Criterion};
use std::hint::black_box;

fn bench_search(c: &mut Criterion) {
    let mut group = c.benchmark_group("search");
    for size in [10usize, 1_000, 100_000] {
        let data: Vec<u32> = (0..size as u32).collect();
        group.bench_with_input(BenchmarkId::new("binary", size), &data, |b, data| {
            b.iter(|| data.binary_search(black_box(&((size - 1) as u32))))
        });
    }
    group.finish();
}

criterion_group!(benches, bench_search);
criterion_main!(benches);
Terminal
cargo bench                                   # run and print results
cargo bench -- --save-baseline before         # save a named baseline
# ...change the code...
cargo bench -- --baseline before              # compare against it

Criterion prints the estimated time with a confidence interval and a verdict such as "Performance has improved" or "No change in performance detected", and writes HTML reports with plots to target/criterion/. Use iter_batched when each iteration needs a fresh input (for example sorting, which mutates), so the cost of building the input is not counted.

divan is a newer alternative with attribute-based benchmarks (#[divan::bench]), generic parameters, and built-in allocation counting. For an exact count of CPU instructions, which is stable even on noisy machines such as CI runners, look at iai-callgrind, which runs the benchmark under Valgrind. Nightly Rust also has the built-in #[bench] attribute, but it is unstable and not recommended.

Pitfalls checklist

  1. Benchmarking a debug build. Always cargo bench or cargo run --release.
  2. No black_box. The optimizer removes the work or precomputes it. Check suspiciously fast results, such as under a nanosecond for a loop.
  3. Measuring setup. Building the input vector inside the timed closure measures allocation, not your algorithm.
  4. One sample. Variation between runs of the same code is often 5 to 20 percent. Look at the distribution.
  5. Unrepresentative data. Sorted or tiny inputs can hide branch-misprediction and cache effects. Benchmark with realistic sizes and orderings.
  6. A noisy machine. Close other programs, plug in a laptop, and avoid shared CI runners for tight comparisons (use instruction counts there).
  7. Micro-benchmark tunnel vision. A function that is 20 percent faster alone may not matter in the real program. Profile first (see Performance and Optimization) and benchmark the hot spot it finds.
  8. Trusting timing without understanding. If the result is surprising, look at the generated assembly or count the work (as above) before believing it.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why should the input of a benchmarked function be passed through black_box?

    black_box hides the value from the optimizer. Without it a call with a literal argument may be folded to a constant, so you would time nothing.
  2. 2. Why prefer the median to the mean when summarizing timing samples?

    Occasional scheduling hiccups create large outliers. The mean moves with them; the median stays near the typical value.
  3. 3. What does calibration in a benchmark harness do?

    Very short operations are below the clock's resolution. Calibration scales the batch size until one sample takes a measurable time such as a millisecond.
  4. 4. Which setting in Cargo.toml lets criterion provide its own main for a bench target?

    harness = false disables the default test harness for that [[bench]] target so criterion_main! can supply main.
  5. 5. You need a stable comparison on a noisy CI machine. Which approach helps most?

    Instruction counts (for example with iai-callgrind) and work counters do not depend on what else the machine is doing, so they are repeatable.

Summary

  • Benchmark in release mode, with warm-up, calibrated batch sizes, many samples and a baseline to compare against.
  • Use Instant for monotonic timing; summarize with the median and spread, and flag outliers with the IQR rule.
  • Wrap inputs and outputs in std::hint::black_box so the optimizer cannot delete or precompute the work.
  • Count deterministic work (comparisons, allocations, instructions) to explain or double-check timings.
  • criterion and divan automate the method and report significance; iai-callgrind gives stable instruction counts for CI.
  • Profile first, benchmark the hot spot, and distrust results you cannot explain.

Next: Advanced Trait Techniques