Skip to content
Tutorials Center
My learning

Topics

Lesson 171 of 201 ยท JVM and Advanced Topics

Benchmarking with JMH

Measure small pieces of Java code correctly. Learn why naive timing fails, then use JMH with @Benchmark, @State, @Param, Blackhole, forks and warm-up.

"Which is faster, String.format or concatenation?" "Does a HashMap beat a sorted array for 50 elements?" "Is that stream slower than the loop?" These questions come up in every code review, and the usual answer is a quick test with System.nanoTime(). Unfortunately the answer you get is very often wrong. The JVM warms up, inlines, removes code whose result is unused, folds constants and moves work between threads. A hand-made timing loop measures all of that noise along with, or instead of, your code.

JMH (the Java Microbenchmark Harness) is the tool built by the JDK's own performance engineers to avoid these traps. It is the standard way to benchmark small pieces of Java code. In this lesson you'll learn why naive benchmarks lie, how a good harness works (with a small one you can run here), and how to write real JMH benchmarks. It is the sequel to the measurement advice in Performance and the profiling tools of Profiling with JFR and VisualVM.

When to use a microbenchmark

A microbenchmark measures one small operation in isolation: a parsing routine, a data structure operation, a serialization method. It is the right tool when you are choosing between two implementations of a hot piece of code, or guarding against a performance regression in a library.

It is the wrong tool for "is my application fast?". Whole-system questions need load tests and a profiler, because real performance depends on caches, the database, the network and GC pressure that a microbenchmark deliberately removes. Never optimize a method just because a microbenchmark says it is faster, unless a profile says that method matters.

Why hand-made benchmarks lie

Here are the traps, in the order people usually fall into them:

  1. Warm-up. The first runs are interpreted. The JIT compiler later compiles hot code, and compiled code can be 10 to 100 times faster. Timing from the start of the program measures the wrong phase.
  2. Dead-code elimination. If the result of a computation is never used, the JIT may skip the computation. Your benchmark then times an empty loop and reports a wonderful speed.
  3. Constant folding. If inputs are compile-time constants, the compiler can compute the answer once, at compile time. Math.sqrt(144.0) costs nothing when the compiler knows the input.
  4. Loop effects. A loop inside a benchmark can be unrolled, vectorized or hoisted, so the cost per iteration changes depending on the loop count. Code in a loop also optimizes differently from the same code called once from elsewhere.
  5. Profile pollution. The JIT specializes code based on what it has seen. If one JVM runs benchmark A and then benchmark B, B may be compiled using A's habits. Results then depend on the order of benchmarks.
  6. Noise. Garbage collection, other processes, CPU frequency scaling and a laptop's thermal throttling add random variation. One measurement tells you almost nothing.
  7. Unrealistic data. Sorting a sorted array, or searching for the same key every time, measures the CPU cache and the branch predictor, not your algorithm.

JMH addresses each of these: it warms up, hands you a Blackhole to consume results, keeps inputs non-constant through @State objects, runs each benchmark in a fresh JVM fork, repeats measurements and reports an error margin.

flowchart TD
    A["JMH starts a fork<br/>(fresh JVM)"] --> B["Warm-up iterations<br/>(not recorded)"]
    B --> C["Measurement iterations<br/>(recorded)"]
    C --> D["Next fork"]
    D --> B
    C --> E["Combine all forks:<br/>score and error"]

Each benchmark runs in several forks, and each fork does warm-up first. This isolates benchmarks from each other.

A tiny harness you can run here

JMH is a library with an annotation processor and runs for minutes, so it can't run in this page. But the principle is easy to demonstrate. The harness below does the three things a hand-made benchmark must: warm up, take several measurements and report the median, and always consume the result (sink). It first checks that both versions give the same answer, then compares string building with += against StringBuilder at three sizes:

Java A careful hand-made harness

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Notice what the program prints and what it leaves out. The exact microseconds differ on every machine and every run, so they are not shown. The stable, trustworthy conclusions are comparisons: the builder is faster, and the gap grows with the input size, because += copies the whole string every time (quadratic work) while the builder appends in place (linear). Testing several sizes is itself a best practice: one size can hide how an algorithm scales.

How JMH is structured

A JMH benchmark is a normal class with annotations, and the harness finds methods by reflection. Our second example shows that idea in miniature: it declares its own @Bench annotation, finds annotated methods, warms each up and times it. The real JMH adds code generation, forks, many modes and statistical output, but the architecture is the same:

Java Finding and running annotated benchmarks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

This is the same mechanism as in Annotations and Reflection. JMH uses an annotation processor instead of reflection at run time: at compile time it generates a separate class per benchmark with tuned timing loops, so the harness's own overhead stays small and predictable.

Writing a real JMH benchmark

Add the dependencies to your build: org.openjdk.jmh:jmh-core and, as an annotation processor, org.openjdk.jmh:jmh-generator-annprocess. The easiest start is the Maven archetype jmh-java-benchmark-archetype, which produces a project whose mvn package builds a self-contained benchmarks.jar. Then a benchmark class looks like this (read-only: it needs the JMH jar):

Java
import java.util.*;
import java.util.concurrent.TimeUnit;
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.infra.Blackhole;

@State(Scope.Benchmark)
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Warmup(iterations = 5, time = 1)
@Measurement(iterations = 5, time = 1)
@Fork(2)
public class LookupBenchmark {

    @Param({"10", "1000", "100000"})
    int size;

    List<Integer> list;
    Set<Integer> set;
    int missing;

    @Setup
    public void setUp() {
        list = new ArrayList<>();
        set = new HashSet<>();
        for (int i = 0; i < size; i++) {
            list.add(i);
            set.add(i);
        }
        missing = -1;                 // the worst case for the list: not found
    }

    @Benchmark
    public boolean listContains() {
        return list.contains(missing);   // returning the value consumes it
    }

    @Benchmark
    public boolean setContains() {
        return set.contains(missing);
    }

    @Benchmark
    public void sumAll(Blackhole blackhole) {
        long total = 0;
        for (int value : list) {
            total += value;
        }
        blackhole.consume(total);        // several results: hand them to a Blackhole
    }
}

Run it with java -jar target/benchmarks.jar LookupBenchmark and JMH prints a table in this shape (illustrative numbers; yours will differ):

TEXT
Benchmark                      (size)  Mode  Cnt        Score      Error  Units
LookupBenchmark.listContains       10  avgt   10        6.1  ยฑ    0.2    ns/op
LookupBenchmark.listContains     1000  avgt   10      410.3  ยฑ    9.8    ns/op
LookupBenchmark.listContains   100000  avgt   10    41200.0  ยฑ 1100.0    ns/op
LookupBenchmark.setContains        10  avgt   10        4.9  ยฑ    0.1    ns/op
LookupBenchmark.setContains      1000  avgt   10        5.2  ยฑ    0.2    ns/op
LookupBenchmark.setContains    100000  avgt   10        6.4  ยฑ    0.3    ns/op

Read it as: the list lookup grows with the size (linear), while the set lookup stays nearly flat (constant time), and Error is the confidence interval around each score. If two scores' error ranges overlap, you cannot claim one is faster.

The annotations you need

Annotation Purpose
@Benchmark Marks a method as a benchmark. JMH calls it repeatedly.
@State(Scope.Benchmark / Thread / Group) A class holding data shared by all threads, or one instance per thread. Fields that are not final stop constant folding.
@Setup / @TearDown Prepare and clean up. Level.Trial runs once per fork, Level.Iteration per iteration, Level.Invocation per call (use rarely: it adds overhead).
@Param Runs the benchmark for each listed value, perfect for input sizes.
@BenchmarkMode Throughput (operations per time), AverageTime (time per operation), SampleTime (distribution and percentiles), SingleShotTime (cold start).
@OutputTimeUnit Unit of the reported numbers.
@Warmup, @Measurement Number and length of warm-up and measured iterations.
@Fork Number of fresh JVMs. Use at least 1, and 2-3 for reliable numbers. @Fork(0) is for debugging only.
@Threads Run the benchmark with several threads at once.

Keeping the benchmark honest

  • Always consume results. Return the value from the method, or pass it to a Blackhole. A benchmark method that computes and discards is the classic dead-code mistake.
  • Keep inputs out of constants. Read input data from @State fields set in @Setup, not from static final constants or literals, or the compiler may fold the work away.
  • No loops inside @Benchmark unless you are measuring the loop itself. JMH already loops around your method; add manual loops and you measure the compiler's loop optimizations. If you need a batch, use @OperationsPerInvocation.
  • Compare equal work. Both versions must produce the same result. Add a correctness check in @Setup or in a unit test (see Unit Testing).
  • Use realistic and varied data. Test several sizes with @Param, and use random but seeded inputs where order or distribution matters.
  • Look at the allocation rate with the -prof gc profiler: java -jar benchmarks.jar -prof gc adds bytes allocated per operation, which often explains a difference in speed. Other profilers (-prof stack, -prof async, -prof perfasm on Linux) show where the time goes inside a benchmark.
  • Control the machine. Close other programs, avoid laptop battery mode, and run the same benchmark several times. Never compare numbers from different machines.
  • Mind the question. The result "A is 3% faster" in a microbenchmark is rarely worth a more complicated design.

If you only need a one-off measurement in a unit-test-like setting, a careful harness like the one in this lesson is fine. As soon as the number drives a decision, a published claim or a regression test, use JMH. It exists because experts got these measurements wrong for years.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is a result computed in a benchmark but never used a problem?

    Dead-code elimination is allowed to drop work whose results cannot be observed. Returning the value, or passing it to a Blackhole, makes it observable.
  2. 2. What is the purpose of @Fork in JMH?

    JIT compilation depends on what code has already run in the JVM. Forks isolate benchmarks from each other and add a measure of run-to-run variation.
  3. 3. Where should benchmark input data normally come from?

    Constants and literals can be folded by the compiler. State fields initialized at runtime hide the values from the optimizer, which is closer to real use.
  4. 4. JMH reports listContains 410.3 ยฑ 9.8 ns/op and setContains 5.2 ยฑ 0.2 ns/op. What can you conclude?

    The scores are far apart compared to their error margins. If two intervals overlap, you cannot claim a difference.
  5. 5. Which question is a microbenchmark LEAST suited to answer?

    Whole-system behavior depends on I/O, caching, contention and GC under realistic load, which isolated microbenchmarks deliberately remove. Use load tests and profilers for that.

Summary

  • Hand-made timing loops are misleading because of warm-up, dead-code elimination, constant folding, loop optimizations, profile pollution and noise.
  • JMH handles these with generated benchmark code, warm-up iterations, separate JVM forks and statistics with error margins.
  • Write @Benchmark methods, keep data in @State and set it up in @Setup, sweep sizes with @Param, and always return results or pass them to a Blackhole.
  • Read the Score ยฑ Error: overlapping intervals mean no proven difference. Use -prof gc to see allocations.
  • Microbenchmarks compare small pieces of code; use load tests and profilers (see JFR) for whole applications.
  • When a quick measurement is enough, use a careful harness: warm up, repeat, take the median, use the result, and compare ratios, not raw times.

Next: Project: Number Guessing Game, where you put what you've learned into a complete program.