Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 40 ยท vmap and Vectorization

vmap with Conditionals and Its Limits

Understand why Python if fails under vmap, how lax.cond becomes select, why where can produce NaN gradients, and what batched while_loop really costs.

A single-example function often contains a decision: "if the value is negative, do this, otherwise do that." Under vmap the function runs on a whole batch at once, and different examples may want different branches. A batch has no single answer to "is x negative?", so the rules for control flow change.

This lesson explains what happens to if, jnp.where, lax.cond, lax.switch and lax.while_loop when they are vmapped, a classic trap with NaN gradients, and the other limits worth knowing before you batch a function.

The plain blocks are real JAX. The runnable blocks reproduce the behavior (both branches evaluated, a loop running until the slowest example finishes) with NumPy, so you can see why the rules exist.

Python if does not work on batched values

Inside vmap (and jit, and grad), an array argument is a tracer: a placeholder that knows its shape and dtype but not its value. A Python if needs one definite True or False, and a tracer cannot provide it.

Python
import jax, jax.numpy as jnp

def relu_like(x):
    if x > 0:               # fails: the value of x is not known while tracing
        return x
    return 0.0

jax.vmap(relu_like)(jnp.array([1., -2., 3.]))
# raises a TracerBoolConversionError

There are two fixes, depending on what the branches look like.

Fix 1: jnp.where for cheap branches

When both branches are simple expressions, compute both and select per element:

Python
def relu_like(x):
    return jnp.where(x > 0, x, 0.0)

print(jax.vmap(relu_like)(jnp.array([1., -2., 3.])))   # [1. 0. 3.]

where(cond, a, b) evaluates a and b for every element and then picks. That is fine when both are cheap and safe, and it is the fastest option on accelerators because there is no branching at all.

Fix 2: lax.cond, which becomes select under vmap

jax.lax.cond(pred, true_fn, false_fn, *operands) is the structured if. For a single scalar predicate it runs only the chosen branch. When pred is batched (different per example), JAX cannot run one branch for the whole batch, so under vmap it quietly converts the cond into a select and evaluates both branches for everyone.

Python
def f(x):
    return jax.lax.cond(x > 0,
                        lambda v: jnp.sqrt(v),    # true branch
                        lambda v: -v,             # false branch
                        x)

print(jax.vmap(f)(jnp.array([4., -9., 16.])))     # [ 2.  9.  4.]

The practical rule: a cond is cheap and skips work only when its predicate is not batched. With a batched predicate, plan on paying for both sides. lax.switch (several branches chosen by an integer index) follows the same rule.

Python Both branches run for every example (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The counters show that each branch ran on all four values even though only two examples wanted each. For two cheap branches that is fine. If one branch is an expensive network and the other a constant, batching a data-dependent cond can erase the saving you expected.

The NaN gradient trap with where

jnp.where selects values, but the gradient flows through both branches. If the unselected branch produces NaN or inf (the square root of a negative number, a division by zero), the gradient of the result can become NaN even though the forward value looked fine.

Python
def safe_sqrt(x):
    return jnp.where(x > 0, jnp.sqrt(x), 0.0)

print(safe_sqrt(-1.0))              # 0.0
print(jax.grad(safe_sqrt)(-1.0))    # nan   <- surprising

The fix is the double where trick: also sanitize the input of the risky branch so it never sees a bad value.

Python
def safe_sqrt(x):
    ok = x > 0
    safe_x = jnp.where(ok, x, 1.0)          # a harmless value where x is bad
    return jnp.where(ok, jnp.sqrt(safe_x), 0.0)

print(jax.grad(safe_sqrt)(-1.0))            # 0.0
Python Why a masked sqrt still leaks NaN (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The naive gradient contains nan at the masked position because 0 * nan is nan. Sanitizing the input first leaves only finite numbers, so the product is a clean 0. The same trick applies to log, division, arccos and any function with a restricted domain.

Loops: while_loop under vmap

lax.while_loop(cond_fn, body_fn, init) repeats until cond_fn is false. If the loop condition depends on batched values, each example wants a different number of iterations. JAX handles this by running the loop until the slowest example finishes, and masking updates for the examples that are already done. Their results are correct, but you pay for the longest one.

Python A batched loop runs for the slowest example (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The batched loop ran 111 iterations (set by the number 27), although the total useful work is much smaller. Short examples sat masked while the long one finished. Similar waste is why you should avoid mixing very different run lengths in one batch, or sort by expected length.

Other limits to remember

Limit What happens Remedy
Output shape depends on values Error: shapes must match across the batch Return a fixed-size array plus a mask or a count
print / append in the function Runs once at trace time with tracers jax.debug.print for per-example output
Mapped sizes disagree Error naming the inconsistent sizes Fix in_axes or slice the data
Random keys are shared by accident Every example gets identical noise Split a key per example (see PRNG Keys)
Huge batch Memory use grows with batch size lax.map or micro-batches

For debugging values inside a mapped function, use jax.debug.print("x = {}", x); it prints actual values at run time (once per example, order not guaranteed).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does a Python if on an array value fail inside vmap?

    During tracing the function sees placeholders with shape and dtype but no concrete value, so Python cannot branch on them.
  2. 2. What happens to lax.cond when its predicate is batched under vmap?

    A batch may contain examples wanting different branches, so JAX computes both and selects the right result per example.
  3. 3. How do you avoid NaN gradients from jnp.where(x > 0, jnp.sqrt(x), 0.0)?

    The unselected branch still takes part in the gradient. Feeding it a safe value means no NaN appears to be multiplied by zero.
  4. 4. A vmapped while_loop with different iteration counts per example runs for how long?

    The loop continues while any example is unfinished, masking updates for the finished ones.

Summary

  • Python if on batched values fails under vmap; use jnp.where or lax.cond.
  • A batched lax.cond or lax.switch becomes a select: every branch runs for every example.
  • jnp.where can leak NaN into gradients; sanitize the risky branch's input with a second where.
  • A batched lax.while_loop runs until the slowest example is done.
  • Output shapes must be identical across examples; use masks for variable-length results.

Next: PRNG Keys and split: Random Numbers the JAX Way