Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 40 ยท jit and XLA

jit with Control Flow and Recompilation

Write branches and loops that compile under jax.jit: jnp.where, lax.cond, lax.scan and fori_loop, plus how to spot and avoid accidental recompilation and padding tricks.

Control flow is where jit surprises newcomers. A Python if or for is evaluated while tracing, using Python values; compiled code has to handle every possible runtime value with one fixed program. This lesson shows the tools that bridge the gap, and how to keep the compilation cache from thrashing.

You met the low-level versions in Low-level Operations and Control Flow with lax; here the focus is on using them well with jit. JAX cannot run in the in-browser runtime (see What Is JAX?), so JAX code is read-only and the editors use NumPy and plain-Python stand-ins that compute the same results.

Option 1: compute both, select with where

For simple branches the cheapest approach is to evaluate both sides and pick per element:

Python
@jax.jit
def leaky_relu(x, slope=0.01):
    return jnp.where(x > 0, x, slope * x)

There is no branch at all, so it vectorizes and compiles trivially. Both sides are always computed, which matters in two cases:

  • a side is expensive, then use lax.cond;
  • a side produces nan or inf for inputs where it is not selected, which can poison gradients even though the forward value is fine. The "double where" trick fixes it:
Python
def safe_log(x):
    ok = x > 0
    safe_x = jnp.where(ok, x, 1.0)       # replace bad inputs before the risky op
    return jnp.where(ok, jnp.log(safe_x), 0.0)
Python Stand-in: where evaluates both sides

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Option 2: lax.cond for a real branch

lax.cond(pred, true_fn, false_fn, *operands) compiles both branch bodies into the program but executes only one, chosen at run time by a traced boolean:

Python
from jax import lax

@jax.jit
def f(x):
    return lax.cond(x.sum() > 0,
                    lambda v: v * 2.0,       # taken when the predicate is True
                    lambda v: v - 1.0,
                    x)

Both branches must return the same structure, shapes and dtypes. Use cond when the branches do very different amounts of work. Under vmap, cond becomes select and computes both branches, as covered in vmap with Conditionals and Its Limits. For more than two branches there is lax.switch(index, [f0, f1, f2], *operands).

Option 3: loops with fori_loop, while_loop and scan

A Python for loop under jit is unrolled: the body is traced once per iteration, so a 1,000-step loop makes a 1,000-times larger program and compiles slowly. Structured loops compile the body once.

Python
# fixed trip count, no per-step outputs
def body(i, acc):
    return acc + i * 2.0
total = lax.fori_loop(0, 100, body, 0.0)

# data-dependent trip count
def cond_fn(state):
    x, steps = state
    return x > 1.0
def body_fn(state):
    x, steps = state
    return x / 2.0, steps + 1
x, steps = lax.while_loop(cond_fn, body_fn, (1000.0, 0))

# running state AND per-step outputs: the workhorse
def step(carry, x):
    carry = carry + x
    return carry, carry                     # (new carry, output for this step)
final, running_sums = lax.scan(step, 0.0, jnp.arange(5.0))

scan is the one to learn first. It loops over the leading axis of its inputs, threads a carry through the steps and stacks the outputs. Recurrent networks, simulations and cumulative statistics are all scans. Reverse-mode differentiation works through scan and fori_loop with a fixed trip count; while_loop supports forward mode only because the number of iterations is not known when the backward pass is planned.

Python Stand-in: scan written as a Python loop

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python Stand-in: while_loop and fori_loop

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Recompilation: the silent slowdown

A jitted function recompiles whenever the cache key changes: new shapes, new dtypes, new static values, or a changed Python structure of the arguments. In a training loop this can destroy performance without any error. Typical causes:

Cause Fix
The last batch is smaller drop it, or pad to the full batch size and mask
Sequences of varying length pad to a few bucket sizes (64, 128, 256...)
A Python float passed as a static argument that changes pass it as a normal (traced) argument instead
A new lambda or partial created every iteration create the jitted function once, outside the loop
Mixing float32 and float64 or Python scalars use consistent dtypes

You can watch compilations with jax.config.update("jax_log_compiles", True), which prints a line each time a function is compiled; one line per iteration means something is changing.

Padding and masking

Python
def pad_to(x, n):
    pad = n - x.shape[0]
    return jnp.pad(x, (0, pad)), jnp.arange(n) < x.shape[0]    # data, valid-mask

@jax.jit
def masked_mean(x, mask):
    return jnp.sum(jnp.where(mask, x, 0.0)) / jnp.sum(mask)

All inputs now have one shape, so one compiled program serves every length up to n.

Python Stand-in: bucketed padding keeps one program

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Branching inside jit

jnp.where Both sides, pick per element
lax.cond One branch runs
lax.scan Loop with a carry
1 / 2

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does lax.cond execute at run time?

    cond compiles both bodies but runs one; under vmap it degrades to select and runs both.
  2. 2. Why is a Python for loop with 1000 iterations a problem under jit?

    Each iteration is traced separately. lax.scan or fori_loop compile the body once.
  3. 3. Which loop primitive does NOT support reverse-mode differentiation?

    while_loop has an unknown trip count, so only forward mode is available.
  4. 4. A training loop recompiles every step. A likely cause?

    New shapes, dtypes or static values produce new cache keys.
  5. 5. What does the double-where trick protect against?

    Replacing bad inputs before the risky operation keeps every intermediate finite.

Summary

  • Use jnp.where for cheap element-wise branches, lax.cond / switch for real branches.
  • Use lax.scan (carry plus stacked outputs), fori_loop and while_loop instead of long Python loops.
  • while_loop is forward-mode only; scan and fixed fori_loop support reverse mode.
  • Recompilation is triggered by new shapes, dtypes or static values; pad, mask and reuse jitted functions.
  • jax_log_compiles shows you when compilation happens.

Next: jit Pitfalls and Side Effects