Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 40 ยท jit and XLA

Async Dispatch and Benchmarking JAX Code

Time JAX code the right way: understand asynchronous dispatch, use block_until_ready, separate compile time from run time, and avoid common benchmarking mistakes.

You write result = f(x) and time it: 0.0002 seconds, wonderful. Then you print result and the program pauses for two seconds. Nothing is wrong with your clock; JAX returned before the work finished. Benchmarking JAX correctly means understanding asynchronous dispatch, compilation time and warm-up. This lesson gives you a reliable recipe.

JAX cannot run in the in-browser runtime (see What Is JAX?), so the JAX code is read-only. The editors use Python threads to model asynchronous dispatch, and NumPy timing to show the measurement habits.

Asynchronous dispatch

When you call a JAX function on an accelerator, the call enqueues the computation and returns immediately with an array object that is a future: a handle to a result that is still being computed. Python carries on while the device works. This lets the host prepare the next operation while the GPU or TPU is busy.

Python
import jax, jax.numpy as jnp, time

x = jnp.ones((4000, 4000))
t0 = time.perf_counter()
y = x @ x                               # returns at once
print("dispatch:", time.perf_counter() - t0)       # tiny, misleading

y.block_until_ready()                   # wait for the device
print("total:", time.perf_counter() - t0)          # the real time

Anything that needs the actual value forces a wait: print(y), np.asarray(y), float(y[0]), y.item(). For timing, call .block_until_ready() on the result (or jax.block_until_ready(tree) for a pytree of outputs).

sequenceDiagram
    participant P as Python
    participant D as Device
    P->>D: enqueue x @ x
    D-->>P: future (returns at once)
    P->>D: enqueue next op
    Note over D: computing...
    P->>D: block_until_ready()
    D-->>P: result ready

Python runs ahead; block_until_ready marks the point where it waits.

Python Stand-in: a queue that works while Python moves on

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Compile time versus run time

The first call of a jitted function includes tracing and compilation, which can take far longer than the computation. Later calls only run the executable. A fair benchmark therefore has three steps:

  1. Warm-up call: run once (and block) so compilation is excluded.
  2. Timed calls: run several times, blocking each result.
  3. Report the mean or best of many runs, not a single sample.
Python
f = jax.jit(my_function)
f(x).block_until_ready()                 # warm-up: compile

import timeit
n = 100
t = timeit.timeit(lambda: f(x).block_until_ready(), number=n) / n
print(f"{t * 1e3:.3f} ms per call")

In a notebook, %timeit f(x).block_until_ready() does the repeats for you.

If you want to know the compilation time itself, time the first call separately, or use the ahead-of-time API: jax.jit(f).lower(x).compile() compiles without running.

Python Stand-in: warm-up, repeats and best-of-n

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Other measurement pitfalls

  • Timing the dispatch, not the work: always block.
  • Including host-to-device transfer: jnp.array(numpy_array) copies data to the device. Move inputs once, before the timer, with jax.device_put.
  • Changing shapes between runs triggers recompilation, so the benchmark measures the compiler (jit with Control Flow and Recompilation).
  • Too small a workload: if a call takes microseconds, Python overhead dominates. Compare on realistic sizes.
  • Comparing against NumPy unfairly: NumPy runs synchronously on the CPU; JAX numbers on CPU may be similar, while GPU and TPU numbers can be 10 to 100 times better for large arrays. Small arrays are often faster in NumPy.
  • float64 versus float32: JAX defaults to float32; using float64 on a GPU can be much slower.
  • Constant folding: if a jitted function depends on no arguments, XLA may compute the result at compile time and the "run" measures nothing. Feed it real arguments.

Understanding a profile

For more than wall-clock times, JAX integrates with profilers:

Python
jax.profiler.start_trace("/tmp/jax-trace")      # then view in TensorBoard or Perfetto
f(x).block_until_ready()
jax.profiler.stop_trace()

A profile shows kernels on the device timeline, memory use, and gaps where the device waited for Python or for data. Large gaps usually mean the input pipeline or Python code is the bottleneck, not the math. Use jax.profiler.annotate_function or TraceAnnotation to label regions.

Async dispatch has benefits too

Because Python runs ahead, a training loop like this keeps the accelerator busy:

Python
for batch in data:
    params, loss = step(params, batch)       # enqueued, not waited for
    losses.append(loss)                      # keeps the future, no sync
print(jnp.stack(losses))                     # one sync at the end

Calling float(loss) or print(loss) on every iteration forces a wait each time and idles the device while Python formats the number. Log every N steps, or accumulate and read the values in batches.

Asynchronous execution also changes where errors surface: a failure in the enqueued work can be raised later, at the next call that touches the result. When debugging, set jax.config.update("jax_disable_jit", True) or block after the suspect call.

Rule of thumb: never time a JAX call without block_until_ready(), and never trust the first call.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why can y = f(x) appear to finish instantly?

    The computation is enqueued; waiting happens when you block or read the value.
  2. 2. Which call correctly waits for a JAX array result before stopping the clock?

    block_until_ready() waits for the device to finish computing that array.
  3. 3. Why include a warm-up call before timing a jitted function?

    After the warm-up, repeated calls measure only the compiled executable.
  4. 4. What is wrong with calling float(loss) every training step?

    Reading a value waits for the computation, so asynchronous overlap is lost.
  5. 5. A benchmark's input array is built with jnp.array(np_array) inside the timer. What is measured too?

    Use jax.device_put before the timer so only the computation is measured.

Summary

  • JAX dispatches work asynchronously; calls return futures.
  • Time with .block_until_ready() and repeat the measurement.
  • Warm up first: the initial call includes compilation.
  • Keep transfers, shape changes and tiny workloads out of benchmarks.
  • Avoid syncing every step in training loops; profile with jax.profiler for the full picture.

Next: Inspecting jaxprs, StableHLO and Compiled Code