Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 40 ยท Parallelism and Performance

Mixed Precision, Buffer Donation and Debugging

Speed up JAX programs with bfloat16 and float16, reuse memory with donate_argnums, and debug jitted code with jax.debug.print, NaN checks and disabling jit.

Once a model trains correctly, three practical topics decide whether it trains well: using cheaper number formats, avoiding unnecessary memory copies, and finding bugs inside compiled code, where ordinary print no longer works the way you expect.

JAX runs only on your own machine, not in this site's editors, so the JAX snippets below are read-only. The runnable blocks use NumPy (which has float16 and float32) to demonstrate the number-format behavior; bfloat16 is emulated by truncating a float32. Printed values come from NumPy.

Number formats: float32, float16 and bfloat16

Format Bits Exponent bits Mantissa bits Range Notes
float32 32 8 23 about 1e-38 to 3e38 the JAX default
float16 16 5 10 about 6e-5 to 65,504 precise but overflows easily
bfloat16 16 8 7 same range as float32 coarse but rarely overflows

bfloat16 keeps float32's exponent range and throws away mantissa precision. Neural networks tolerate low precision much better than they tolerate overflow, which is why TPUs and recent GPUs favor it. It halves memory and often doubles matrix-multiply speed.

Python What float16 and a bfloat16-like format can and cannot represent

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

float16 keeps 1 + 0.001 roughly (1.000977) but overflows 70000 to infinity. The bfloat16-like format survives the large value (rounded to 69632) but loses the 0.001 completely, because it has only 7 mantissa bits. Adding many small numbers one at a time in a 16-bit accumulator drifts or stalls, so keep running sums and reductions in float32.

The standard recipe: mixed precision

  1. Keep master parameters and the optimizer state in float32.
  2. Cast parameters and activations to bfloat16 for the forward and backward passes.
  3. Compute losses, softmax, normalization statistics and reductions in float32.
  4. Update the float32 master copy.
Python
import jax, jax.numpy as jnp

def forward(params, x):
    p16 = jax.tree_util.tree_map(lambda a: a.astype(jnp.bfloat16), params)
    h = jnp.tanh(x.astype(jnp.bfloat16) @ p16["w"] + p16["b"])
    return h.astype(jnp.float32)              # back to float32 for the loss

def loss_fn(params, x, y):
    return jnp.mean((forward(params, x) - y) ** 2)

Gradients arrive in float32 because params are float32 and the astype is differentiable. Flax layers also take dtype= (computation dtype) and param_dtype= (storage dtype) arguments, so nn.Dense(128, dtype=jnp.bfloat16) does the cast for you. With float16 (not bfloat16) you additionally need loss scaling, multiplying the loss so small gradients do not underflow; bfloat16 usually does not.

You can also control matmul internals with jax.default_matmul_precision("highest") when a bfloat16-pass matrix multiply hurts accuracy, and enable float64 with jax.config.update("jax_enable_x64", True) before creating arrays (see arrays and dtypes).

Buffer donation

When you update parameters with new_params = update(params, grads), the old and new parameters exist at once; for a large model that doubles the memory. Donation tells XLA the input buffer is no longer needed, so the output may reuse it.

Python
from functools import partial

@partial(jax.jit, donate_argnums=(0,))
def train_step(params, batch):
    grads = jax.grad(loss_fn)(params, *batch)
    return jax.tree_util.tree_map(lambda p, g: p - 0.01 * g, params, grads)

params = train_step(params, batch)    # always rebind: the old params are invalid now

The rule is strict: after donating an argument you must not use the old array again, and reading it raises an error (on backends that implement donation; the CPU may ignore it with a warning). Donate the state you immediately replace: parameters, optimizer state, large accumulators. Never donate data you reuse, like the next epoch's batch.

Debugging under jit

Inside a jitted function, arguments are tracers, not values, so print(x) shows Traced<ShapedArray(...)>. Use the tools built for the job:

Python
@jax.jit
def f(x):
    y = jnp.log(x)
    jax.debug.print("min x = {m}, any nan = {n}", m=x.min(), n=jnp.isnan(y).any())
    return y.sum()

f(jnp.array([1.0, 0.0, -1.0]))

jax.debug.print is staged into the compiled program and prints at run time with real values, even under jit, vmap and scan. Related tools:

  • jax.debug.breakpoint() pauses with a debugger prompt (inspect values interactively).
  • jax.disable_jit() (a context manager) runs everything op by op in Python, so you can use print, pdb and ordinary exceptions. Slow, but great for finding logic bugs.
  • jax.config.update("jax_debug_nans", True) re-runs the failing operation and raises at the first NaN, pointing to the source line. Turn it off for production runs.
  • jax.config.update("jax_debug_infs", True) does the same for infinities.
  • jax.checking_leaks() or leaked-tracer errors tell you a traced value escaped its function (see jit pitfalls).
  • jax.make_jaxpr(f)(x) shows what was traced (jaxprs and HLO).
Python A tiny NaN hunter (NumPy stand-in for jax_debug_nans)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real NaN hunting works the same way in spirit: find the first operation producing a non-finite value, not the loss where it finally shows up. Common culprits are log(0), sqrt of a negative, 0/0, exp of a large number and the where trap.

A debugging checklist

  1. Run the function with jax.disable_jit() and small inputs.
  2. Check shapes and dtypes with jax.eval_shape(f, x), which traces without computing.
  3. Turn on jax_debug_nans and reproduce.
  4. Add jax.debug.print around the suspect operation.
  5. Only then optimize: measure with proper benchmarking.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is bfloat16 popular for training?

    bfloat16 has 8 exponent bits like float32 and only 7 mantissa bits; networks tolerate the lower precision.
  2. 2. What must you do after calling a function jitted with donate_argnums=(0,)?

    The donated buffer may be overwritten by the output, so the old array is invalid.
  3. 3. What does jax.debug.print do that print does not?

    print runs once at trace time with tracers; jax.debug.print is staged into the program.
  4. 4. Which setting makes JAX raise at the first operation that produces a NaN?

    jax_debug_nans re-runs the failing computation and points to the offending operation.

Summary

  • float32 is the default; bfloat16 trades precision for range and speed; float16 needs loss scaling.
  • Mixed precision keeps master weights and reductions in float32 and computes in bfloat16.
  • donate_argnums lets outputs reuse input buffers; never use a donated input afterward.
  • Use jax.debug.print, jax.disable_jit() and jax_debug_nans instead of plain print under jit.
  • Locate the first non-finite value, not the place where the loss turns to NaN.

Next: Project: Fit a Curve with Gradient Descent