Skip to content
Tutorials Center
My learning

Topics

Lesson 82 of 134 ยท Standard Library

Benchmarking

Learn how to benchmark Ruby code - Benchmark.realtime and measure, the monotonic clock, comparing approaches fairly, counting allocations with GC.stat, warm-up, and profiling tools like stackprof.

"Make it fast" is a request that needs a number. Programmers are famously bad at guessing where a program spends its time: the loop you suspect is often irrelevant, and the innocent-looking line is the bottleneck. Benchmarking measures how long a piece of code takes; profiling tells you where a whole program spends its time or memory. The rule of the trade is: measure first, change one thing, measure again.

Ruby ships a benchmark library, and the core gives you a precise clock and counters for memory allocations. One caveat for this page: timings differ on every run and on every machine, so the examples here never print the times themselves. They print conclusions that hold by a wide margin (such as "the set lookup was faster"), numbers that are always the same (such as how many objects were allocated), or the shape of the result. In your own projects, you will of course print the real numbers.

Measuring time

There are three ways to measure, from the most to the least reliable for timing code:

  1. Process.clock_gettime(Process::CLOCK_MONOTONIC): a clock that only moves forward, unaffected by clock adjustments. Best for measuring durations.
  2. Benchmark.realtime { ... }: wall-clock seconds as a Float; a convenient wrapper around the above.
  3. Benchmark.measure { ... }: a Benchmark::Tms record with CPU user time, system time and real time.
Ruby realtime, measure and the monotonic clock

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

Benchmark.measure splits time into user CPU time (your Ruby code), system CPU time (the kernel working for you) and real (wall clock) time. For code that waits on a network or sleeps, real time is much larger than CPU time. Benchmark is itself a bundled gem in recent Ruby versions: it is installed with Ruby and works in the runner, but a Bundler project should list gem "benchmark" in its Gemfile. The monotonic clock needs no gem at all.

Never measure with Time.now - start. The wall clock can be adjusted while your code runs (a clock sync, or a change of daylight saving time), producing negative or wildly wrong durations. See Time and Date for what Time is good at.

Comparing approaches fairly

The usual question is "which of these two ways is faster?" Several mistakes make the answer meaningless:

  • Measuring once. One run is noisy. Repeat and take the best or the median.
  • Tiny inputs. If the data is small, everything is fast and the difference is lost in noise. Use realistic sizes.
  • Order and warm-up effects. The first run can pay for caching or lazy loading, so warm up before you measure.
  • Garbage collection. A GC pause may land in either measurement; run several rounds.
  • Measuring the wrong thing. Make sure both versions do the same work and return the same result.

Here is a comparison done carefully: finding membership in a big Array versus a Set. The two versions must give the same answers, and the difference is large enough (a linear scan against a hash lookup) that the conclusion is stable.

Ruby Array include? versus Set include?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

This follows the checklist: same results are verified first, input size is realistic, each version runs five times and the best time is kept (the minimum is the least disturbed by noise), and the printed conclusion rests on a big gap rather than a few percent. On your own machine, print array_time and set_time to see the actual seconds. (Complexity is covered in Sets: an array scan is O(n), a set lookup is O(1) on average.)

Benchmark.bm for side-by-side reports

Benchmark.bm runs several labeled blocks and prints a table of user, system, total and real time for each. Printing a table with times is fine on your own machine; here we capture the output and show only its structure.

Ruby Benchmark.bm report structure

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The table has a header line with the columns user system total real and one row per report. The label width argument (8) pads the labels so the columns line up. Redirecting $stdout to a StringIO (see StringIO and Streams) is a standard way to capture output, and always restore it afterwards, preferably in an ensure.

Counting allocations: a deterministic measurement

Time is noisy; object counts are not. Every Ruby object you create costs memory and later garbage-collection work, so counting allocations is a stable way to compare code. GC.stat(:total_allocated_objects) is a counter that only goes up.

Ruby Counting allocated objects

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

s += "x" means s = s + "x", which builds a brand-new string each time (quadratic work for long loops), while s << "x" appends to the one buffer. Allocation counts like these are the reason "build with << or Array#join, not +=" is advice you will hear in every Ruby performance talk. The example also shows that a plain "lit" string literal allocates a new String each time it runs, which is why hot loops sometimes use a frozen constant (see Immutability and freeze). Exact counts vary slightly between Ruby versions, so compare them against each other rather than against a magic number.

Comparing algorithms by counting steps

Sometimes the best "benchmark" does not use a clock at all. If two algorithms differ in how many operations they do, count them. This result is exact and identical on every machine:

Ruby Comparisons made by linear and binary search

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The linear search needs roughly one step per item (O(n)); binary search halves the range each time (O(log n)), about 20 steps for a million items. When complexity is your question, counting steps like this shows the growth directly, and the real timings will agree. The approach scales from toy examples to real code: add a counter, run both versions, compare.

Profiling: finding the real bottleneck

A benchmark compares two pieces of code you already suspect. A profiler finds the culprit in a program you don't understand yet. Ruby profilers are third-party tools, so these snippets are read-only (they are not run here):

Terminal
# read-only example: needs gems that are not available in the runner
gem install stackprof memory_profiler benchmark-ips

# Sampling CPU profile of a script, then a report of the hottest methods
stackprof --cpu --out tmp/profile.dump -- ruby app.rb
stackprof tmp/profile.dump --text --limit 10
Ruby
# read-only example: benchmark-ips reports iterations per second
require 'benchmark/ips'

Benchmark.ips do |x|
  x.report("array")  { numbers.include?(12_345) }
  x.report("set")    { as_set.include?(12_345) }
  x.compare!
end
Ruby
# read-only example: memory_profiler shows what allocates and retains objects
require 'memory_profiler'

report = MemoryProfiler.report { run_expensive_code }
report.pretty_print

benchmark-ips runs each block for a fixed number of seconds (after a warm-up) and reports iterations per second with an error margin, which handles the fairness problems from earlier automatically. stackprof samples the call stack many times a second and reports which methods appear most, so it adds little overhead. memory_profiler tells you which lines allocate the most objects and which survive. For web apps, tools such as Rack Mini Profiler and APM services do the same for requests.

A measurement checklist

  1. Decide what matters. Is it speed, memory, throughput, or latency of one request?
  2. Measure the whole program first to see whether the code in question matters at all.
  3. Use the monotonic clock (or Benchmark.realtime), several rounds, and keep the minimum or median.
  4. Use realistic data sizes and make both versions return the same answer.
  5. Warm up and avoid I/O and printing inside the timed block.
  6. Change one thing at a time, then measure again.
  7. Write down the result and the Ruby version (and whether YJIT was on): the numbers change between versions. See Performance Tips.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which clock is the right choice for measuring how long code runs?

    The monotonic clock only moves forward and is not changed by clock synchronisation, so differences between two readings are reliable durations.
  2. 2. In Benchmark.measure, what do utime and stime represent?

    utime is user CPU time, stime is system CPU time; real is the elapsed wall-clock time, which also includes waiting.
  3. 3. Why is one single timing of a code block a poor basis for a conclusion?

    One run can be disturbed by a GC pause or by cold caches. Several rounds with the minimum or median give a more stable number.
  4. 4. Why use GC.stat(:total_allocated_objects) to compare two versions of code?

    The allocation counter is deterministic for the same code, so it can be compared without noise. Fewer allocations usually means less memory pressure and less GC work.
  5. 5. What is the main job of a profiler such as stackprof?

    A profiler samples the running program and ranks where time goes, so you optimise the real hot spot instead of guessing.

Summary

  • Measure before optimizing; use Process.clock_gettime(Process::CLOCK_MONOTONIC) or Benchmark.realtime, never Time.now differences.
  • Benchmark.measure returns user, system and real time; Benchmark.bm prints a comparison table. benchmark is a bundled gem: list it in your Gemfile.
  • Compare fairly: same results, realistic sizes, several rounds, warm-up, one change at a time.
  • Allocation counts (GC.stat(:total_allocated_objects)) and step counts are deterministic and great for tests and explanations.
  • Profilers (stackprof, memory_profiler) and benchmark-ips are third-party tools; they find bottlenecks and handle statistics for you.

Next: Digest, Base64 and Encoding