Skip to content
Tutorials Center
My learning

Topics

Lesson 113 of 182 ยท Memory and Performance

Benchmarking with BenchmarkDotNet

Learn to measure C# performance correctly - why Stopwatch timings mislead, how BenchmarkDotNet warms up and repeats, memory diagnosers, Params, baselines and how to read the results.

"Is this faster?" is one of the easiest questions to answer wrongly. Wrap two methods in a Stopwatch, run them once, and the first one usually loses simply because it paid for JIT compilation. Run them in a Debug build and you measure code the optimizer never touched. Forget to use the result and the compiler may throw the whole computation away. The numbers look precise, but they describe nothing.

BenchmarkDotNet is the library the .NET team and most of the community use to avoid these traps. It runs each method many times in a separate process, warms it up, removes its own overhead, reports statistics and can count allocations. In this lesson you will see exactly which mistakes it protects you from (using small programs you can run here), how to write and read a real benchmark, and how to decide whether a measured difference is worth caring about. It builds on Performance Basics; the ideas behind allocation numbers are in Garbage Collection.

BenchmarkDotNet is a NuGet package and it launches child processes and uses profiler hooks, so it cannot run in this page's sandbox. Its code is shown read-only. The runnable examples below use only the base library to demonstrate the underlying problems and ideas, and they print results that do not depend on how fast your machine is.

Why a quick Stopwatch test lies

A runtime like .NET does a lot of work behind your back the first time code runs:

  • JIT compilation. Methods are compiled to machine code when first called, so the first call includes compile time (IL and the JIT).
  • Tiered compilation. Hot methods are recompiled with heavier optimization after about thirty calls, so early iterations run slower code than later ones.
  • Debug vs Release. A Debug build disables most optimizations. A benchmark must always be built with -c Release.
  • Dead-code elimination. If a result is never used, the compiler or JIT may remove the computation.
  • Noise. Other processes, CPU frequency changes, caches and the garbage collector all add random time to any single run.

The first of these is easy to see, so let us measure what the measurement itself shows without printing the machine-dependent numbers:

C# The first call is not like the others

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

The checksum is the same on every machine, but the eight timings are not, and the first is often the slowest. That variation is exactly why a single measurement is meaningless. A proper harness repeats the work, throws away warm-up runs, and reports the median and the spread.

What BenchmarkDotNet does for you

A benchmark class holds methods marked with [Benchmark]. The harness runs the work through several phases in a freshly started process:

flowchart TD
    A["Build in Release<br/>(separate project)"] --> B["Launch a new process<br/>per benchmark"]
    B --> C["Jitting: call once so<br/>compilation is not measured"]
    C --> D["Pilot: find how many calls<br/>fit in about 500 ms"]
    D --> E["Warmup: untimed runs until<br/>times settle (tiered JIT done)"]
    E --> F["Actual: 15 or more timed<br/>iterations, minus overhead"]
    F --> G["Statistics: mean, median,<br/>error, StdDev, allocations"]

Each benchmark gets its own process and a fixed pipeline, so one benchmark cannot warm up, fragment or crash another.

Here is the typical shape of a benchmark project. This is read-only because it needs the NuGet package:

C#
// Read-only: needs the BenchmarkDotNet NuGet package (dotnet add package BenchmarkDotNet)
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkRunner.Run<StringJoinBenchmarks>();

[MemoryDiagnoser]
public class StringJoinBenchmarks
{
    private string[] _words = [];

    [Params(10, 1_000)]
    public int Count;

    [GlobalSetup]
    public void Setup() => _words = Enumerable.Range(0, Count).Select(i => "w" + i).ToArray();

    [Benchmark(Baseline = true)]
    public string Concatenate()
    {
        string result = "";
        foreach (var word in _words)
        {
            result += word + ",";
        }

        return result;
    }

    [Benchmark]
    public string UseStringBuilder()
    {
        var sb = new System.Text.StringBuilder();
        foreach (var word in _words)
        {
            sb.Append(word).Append(',');
        }

        return sb.ToString();
    }

    [Benchmark]
    public string UseStringJoin() => string.Join(',', _words);
}

You run it from the command line, always in Release and never from the debugger:

Terminal
dotnet run -c Release

The attributes you will use most:

Attribute Purpose
[Benchmark] Marks a method to measure. Return the result so it counts as used.
[Benchmark(Baseline = true)] Other rows show a ratio relative to this one.
[Params(a, b, c)] Runs every benchmark once per value of a field or property (input sizes).
[GlobalSetup] / [GlobalCleanup] Runs once per benchmark and parameter set, outside the timing.
[IterationSetup] Runs before each iteration. Only for work that must be reset (it makes very short benchmarks inaccurate).
[MemoryDiagnoser] Adds columns for GC collections and allocated bytes.
[SimpleJob(RuntimeMoniker.Net10_0)] Chooses runtime(s); several jobs compare versions side by side.

Reading the results

The output is a table. A typical result for the benchmark above (the numbers are illustrative, yours will differ) looks like this:

TEXT
| Method           | Count | Mean       | Error     | StdDev    | Ratio | Gen0   | Allocated |
|----------------- |------ |-----------:|----------:|----------:|------:|-------:|----------:|
| Concatenate      | 1000  | 612.40 us  |  9.80 us  |  8.70 us  |  1.00 | 3100.0 | 6320 KB   |
| UseStringBuilder | 1000  |  11.90 us  |  0.21 us  |  0.19 us  |  0.02 |    4.6 |   37 KB   |
| UseStringJoin    | 1000  |   5.30 us  |  0.10 us  |  0.09 us  |  0.01 |    2.0 |   17 KB   |

How to read the columns:

  • Mean is the average time per call. Look at it together with Error (half of a 99.9% confidence interval) and StdDev. If the error is a large fraction of the mean, the result is noisy and you cannot trust small differences.
  • Ratio compares to the baseline. A ratio of 0.02 means about fifty times faster.
  • Gen0 / Gen1 / Gen2 show garbage collections per 1,000 operations. More collections mean more pauses in a real application.
  • Allocated is the memory allocated per operation. Allocation is often the real cost, because it triggers GC work that does not show up in a single call's time.

Units matter: ns (nanoseconds), us (microseconds), ms. A change from 12 ns to 9 ns is a 25% improvement that no user will ever notice unless the method runs millions of times per request.

Measuring memory without a library

Allocation counts are deterministic enough to measure yourself. GC.GetAllocatedBytesForCurrentThread returns the total bytes the current thread has allocated, so the difference before and after is the cost of the code in between:

C# Counting allocated bytes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

The exact byte counts depend on the runtime version, so we compare them instead of printing them. s += ... in a loop creates a brand-new string on every iteration (500 strings of growing length), which is why its allocation grows with the square of the input. string.Join computes the final length first and allocates once. The same reasoning is behind StringBuilder, and allocation-free code with spans goes even further.

A tiny honest harness

You should not replace BenchmarkDotNet with your own timing loop, but writing a small one shows what the library automates. This one warms up, repeats, uses the result, and compares medians, not single runs. The comparison asks a question with an overwhelming answer (a linear search through 20,000 items against a hash lookup) so the conclusion is the same on any computer:

C# A miniature benchmark harness

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Notice the checks before measuring: the two versions must produce the same answer, otherwise you are comparing different work. Notice also that the data size is realistic and the probes hit the worst case for the list. A benchmark that tests only a 10-element list would conclude the opposite, which is why [Params] with several sizes is so useful. The reasoning about hash lookups continues in Fast Lookup.

Understanding spread: statistics on samples

Libraries report more than the mean because a few slow iterations (a GC pause, another process) can drag it upward. This example computes the statistics from a fixed set of samples, so you can see how each number behaves:

C# Mean, median and outliers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

One slow sample (38.0) moves the mean from about 10.3 to 13.1 while the median stays at 10.3. BenchmarkDotNet detects such outliers and tells you about them in the output. When you see many, the machine is probably busy: close other programs, plug in the power cable, and rerun.

Rules for trustworthy benchmarks

  1. Release build, no debugger. Debug builds and attached debuggers change everything. BenchmarkDotNet refuses to run an unoptimized build and warns you.
  2. Compare equal work. Verify that both versions return the same result before you compare speeds.
  3. Return or consume results. Returning the value from the [Benchmark] method lets the harness consume it, so the JIT cannot delete the work.
  4. Keep setup out of the measurement. Put building input data in [GlobalSetup].
  5. Test realistic sizes. Use [Params] with small, typical and large inputs. Many "optimizations" help only at one end.
  6. Watch allocations as well as time. Add [MemoryDiagnoser] always.
  7. Isolate the question. Benchmarking a single method with a mock database in the loop measures the mock. Benchmark the code you want to change.
  8. Decide first, then measure. Find the slow part with a profiler or the tools in Debugging and Diagnostics, then benchmark candidate fixes. Micro-benchmarks tell you which of two implementations is better, not which part of your app is slow.

A micro-benchmark result is not a prediction of application speed. Caches behave differently in a real program, the method may be called once per request instead of millions of times, and the JIT may inline it differently in context. Treat results as evidence for a decision, then confirm with an end-to-end measurement.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is a single Stopwatch measurement of a method's first call unreliable?

    The runtime compiles methods on first use and tiered compilation improves them later, so early timings are not representative of steady state.
  2. 2. What does BenchmarkDotNet do before it records the timed iterations?

    The pilot finds a sensible number of invocations per iteration and warmup lets tiered compilation settle, so the actual iterations measure steady-state code.
  3. 3. A result table shows Mean = 100 ns and Error = 80 ns. What should you conclude?

    Error is half of the confidence interval. When it is a large part of the mean, the measurement cannot support conclusions about small differences.
  4. 4. What does [Params(10, 1000)] on a field do?

    Params produces one set of results per value, which shows how an implementation scales with input size.
  5. 5. Why should a [Benchmark] method return its result?

    Unused results may be eliminated by the optimizer. Returning the value gives the harness something to consume, which keeps the measured work real.

Summary

  • Naive timing loops are distorted by JIT compilation, tiered compilation, Debug builds, dead-code elimination and noise.
  • BenchmarkDotNet runs each benchmark in its own Release-mode process through jitting, pilot, warmup and actual stages, and reports mean, error, standard deviation and ratios.
  • Mark methods with [Benchmark], mark one as Baseline = true, use [Params] for input sizes, [GlobalSetup] for preparation and [MemoryDiagnoser] for allocations.
  • Allocation per operation is often more telling than time; GC.GetAllocatedBytesForCurrentThread gives a quick check.
  • Prefer the median and look at outliers when samples are noisy.
  • Check that compared versions give the same result, test realistic sizes and confirm winning micro-optimizations with an end-to-end measurement.

Next: Dates and Times