Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 40 ยท Arrays and NumPy API

Low-level Operations and Control Flow with lax

Learn jax.lax, the low-level API behind jax.numpy: lax.cond, switch, fori_loop, while_loop and scan, with pure-Python stand-ins that show exactly what each does.

jax.numpy is a friendly layer on top of jax.lax, a thinner API whose functions map almost one-to-one onto operations in the XLA compiler. For everyday array math you rarely need lax. But it is essential for one job: control flow that works under jit. A Python for loop gets unrolled when traced, and a Python if cannot look at array values. lax.cond, lax.fori_loop, lax.while_loop and lax.scan solve both problems.

The runnable editors here are pure-Python stand-ins that behave like these functions on ordinary values (JAX cannot run in the in-browser runtime; see What Is JAX?). Real JAX code is read-only.

Why Python control flow is not enough

Python
import jax
import jax.numpy as jnp

@jax.jit
def f(x):
    if x > 0:          # x is a tracer: its value is unknown at trace time
        return x
    return -x

f(1.0)   # raises TracerBoolConversionError

Under jit, x is an abstract placeholder, so Python cannot decide which branch to take. Loops have a different issue: for i in range(1000): x = x * 1.01 works but gets unrolled into 1000 copies of the operation, making compilation slow. The lax primitives keep a single copy of the body in the compiled program.

lax.cond: choose a branch

lax.cond(pred, true_fun, false_fun, *operands) evaluates a boolean and runs one of two functions. Both branches must return the same shape and dtype structure.

Python
from jax import lax

def abs_value(x):
    return lax.cond(x > 0, lambda v: v, lambda v: -v, x)

print(jax.jit(abs_value)(-3.0))    # 3.0

Under vmap, cond turns into select, which computes both branches and picks per element. lax.switch(index, [f0, f1, f2], *operands) generalizes to many branches. For simple element-wise choices, jnp.where(pred, a, b) is usually the neatest tool.

Python Stand-in: cond and switch semantics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

lax.fori_loop and lax.while_loop

lax.fori_loop(lower, upper, body_fun, init_val) runs body_fun(i, state) for i from lower up to (not including) upper and returns the final state. The state ("carry") is passed through each iteration; no hidden mutation. lax.while_loop(cond_fun, body_fun, init_val) repeats while cond_fun(state) is true.

Python
def sum_to(n):
    return lax.fori_loop(0, n, lambda i, acc: acc + i, 0)

sum_to(5)                      # 10

def collatz_steps(n):
    def cond(s): return s[0] != 1
    def body(s):
        v, steps = s
        v = jnp.where(v % 2 == 0, v // 2, 3 * v + 1)
        return v, steps + 1
    return lax.while_loop(cond, body, (n, 0))[1]

Important limits: the carry must keep the same shape and dtype each iteration, and reverse-mode differentiation (grad) of a while_loop is not supported (use fori_loop with static bounds, or scan, when you need gradients).

Python Stand-in: fori_loop and while_loop

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

lax.scan: loops that collect results

scan is the most important loop primitive. It threads a carry through a sequence and also stacks the per-step outputs:

Text
carry, ys = lax.scan(f, init, xs)

where f(carry, x) -> (new_carry, y). It is the functional version of "for each element: update state, emit a value", and it is what recurrent networks, cumulative sums, time-stepping simulations and optimizer loops are built from. Unlike fori_loop, scan supports reverse-mode autodiff well, and it compiles to a single loop.

Python
def step(carry, x):
    carry = carry + x
    return carry, carry          # new carry, output for this step

total, running = lax.scan(step, 0.0, jnp.array([1., 2., 3., 4.]))
# total = 10.0 ; running = [ 1.  3.  6. 10.]
flowchart LR
    C0["carry0"] --> S1["f(carry0, x0)"]
    S1 --> C1["carry1"]
    S1 --> Y1["y0"]
    C1 --> S2["f(carry1, x1)"]
    S2 --> C2["carry2"]
    S2 --> Y2["y1"]

The carry flows from step to step; each step also emits one output, and the outputs are stacked into ys.

Python Stand-in: scan, running sums and a recurrence

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Other useful lax functions

  • lax.map(f, xs): apply f to each slice sequentially, using less memory than vmap (a scan in disguise).
  • lax.dynamic_slice / lax.dynamic_update_slice: slice with indexes that are traced values.
  • lax.stop_gradient(x): treat x as a constant during differentiation.
  • lax.associative_scan: parallel prefix scans for associative operations.
  • lax.conv_general_dilated, lax.reduce_window, lax.dot_general: the primitives behind convolutions and matmuls.

Choosing the right tool

Need Use
Element-wise choice jnp.where
One decision, two computations lax.cond
Fixed number of iterations, final state only lax.fori_loop
Unknown number of iterations (no grad) lax.while_loop
Iteration that also collects outputs, or needs grad lax.scan
Small constant loop count a plain Python for (unrolled)

You will use scan again in jit and XLA when we discuss compile time, and in the projects of this course.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does a Python 'if x > 0' fail inside a jitted function?

    During tracing, x is an abstract placeholder, so Python cannot convert the comparison to a real boolean. Use lax.cond or jnp.where.
  2. 2. In lax.scan(f, init, xs), what must f return?

    f(carry, x) returns the next carry and the output for that step; the outputs are stacked into ys.
  3. 3. Which loop primitive does NOT support reverse-mode differentiation?

    A while_loop has an unknown trip count, so reverse-mode autodiff cannot store intermediate values for it.
  4. 4. What must be true of the carry in fori_loop, while_loop and scan?

    The compiled loop has a fixed type signature, so the carry's structure cannot change.

Summary

  • jax.lax is the low-level API; use it for control flow under jit.
  • cond and switch choose branches; fori_loop and while_loop repeat; scan repeats and collects outputs.
  • The carry must keep a fixed type; while_loop cannot be reverse-differentiated.
  • Prefer jnp.where for element-wise choices and scan for recurrences.

Next: grad Basics: Differentiating Python Functions