Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 40 ยท Parallelism and Performance

lax.scan, Rematerialization and Memory

Compile long loops with lax.scan, trade compute for memory with jax.checkpoint (remat), and understand why backprop stores activations, with runnable NumPy stand-ins.

Two things limit big JAX programs: compile time and memory. Both have standard cures. lax.scan turns a Python loop into a single compiled loop, so a thousand steps compile as fast as one. jax.checkpoint (also called remat, short for rematerialization) trades extra computation for lower memory during backpropagation.

JAX is not available in the live editors here. JAX snippets are read-only; the runnable blocks implement scan and the checkpointing idea in NumPy so you can verify the behavior. Printed numbers come from those NumPy programs.

Why Python loops hurt under jit

Under jit, a Python for loop is unrolled during tracing: a loop of 1,000 iterations puts 1,000 copies of the body in the program. Compilation becomes slow and the program large. lax.scan keeps the body once and loops in the compiled code.

Python
import jax, jax.numpy as jnp
from jax import lax

def step(carry, x):
    new_carry = carry * 0.9 + x          # state update
    output = new_carry ** 2              # per-step result
    return new_carry, output             # (carry, y)

final, outs = lax.scan(step, init=jnp.array(0.0), xs=jnp.arange(5.0))

The contract: scan(f, init, xs) calls f(carry, x) for each slice x of xs along axis 0, and f returns (new_carry, y). The result is the final carry and all ys stacked. The carry must have the same shape and dtype on every step. scan also differentiates: reverse-mode runs the loop backward automatically.

Python Implement scan and use it for a running filter (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

scan is the right tool for recurrent networks, time stepping in simulations and cumulative computations. For plain element-wise work over a batch, prefer vmap; for iterating while a condition holds, see lax.while_loop in lax operations.

Useful options

  • length=n runs n steps when there is no xs.
  • reverse=True scans from the last element to the first.
  • unroll=k unrolls k iterations per compiled loop step, which can speed up small bodies.
  • xs and ys may be pytrees: every leaf is sliced along axis 0.

Why training uses so much memory

Reverse-mode differentiation needs intermediate values from the forward pass. A network with L layers and activations of size A stores about L * A numbers until the backward pass reaches each layer. Long sequences, deep networks and large batches make that the dominant cost. The parameters are often a small share of the total.

jax.checkpoint: recompute instead of store

Wrapping a function in jax.checkpoint (alias jax.remat) tells JAX: do not save this function's internal activations; recompute them when the backward pass needs them. You keep only the function's inputs, at the price of running its forward computation twice.

Python
@jax.checkpoint
def block(x, w):
    return jnp.tanh(x @ w)

def deep_net(x, weights):
    def body(h, w):
        return block(h, w), None
    h, _ = lax.scan(body, x, weights)    # weights stacked along axis 0
    return h

Scanning a checkpointed block over stacked weights is a standard pattern: memory grows with the number of blocks' inputs rather than with every intermediate inside them, and compile time stays flat as depth grows.

The simulation below counts stored values for a chain of layers with plain backprop versus checkpointing every k-th layer.

Python Memory versus recomputation (counting simulation)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With k near the square root of the depth, memory shrinks dramatically while compute rises by roughly one extra forward pass (about a third more total work for a typical training step). That is the classic trade: buy memory with FLOPs.

Controlling what to save

jax.checkpoint accepts a policy. For example, jax.checkpoint_policies.dots_with_no_batch_dims_saveable keeps the outputs of matrix multiplications (expensive to recompute) and rematerializes cheap element-wise operations. Choose a policy when a profile shows that full recomputation is too slow.

When to reach for each tool

Problem Tool
Compile takes minutes because of a long loop lax.scan
Out of memory during grad of a deep model jax.checkpoint
Out of memory because batch is too large Smaller batch, gradient accumulation, or sharding
Loop count depends on data lax.while_loop (no reverse-mode gradient)

Common mistakes

  • Changing the carry's shape or dtype between steps: scan requires a fixed carry type. Cast the initial carry (jnp.float32(0), not 0).
  • Using a Python for loop inside jit for hundreds of steps: compile time explodes.
  • Putting side effects in the scan body (printing, mutation): the body is traced once and does not run per step in Python. Use jax.debug.print if you need to see values.
  • Checkpointing everything: remat costs time. Apply it to the repeated large block, not the whole program.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does lax.scan(f, init, xs) require f to return?

    f maps (carry, x) to (new_carry, y); the carry keeps its shape and dtype, the ys are stacked.
  2. 2. Why is a Python for loop of 1,000 steps slow to compile under jit?

    Tracing runs the Python loop, so the body is recorded once per iteration.
  3. 3. What does jax.checkpoint (remat) trade?

    Activations are not stored; they are recomputed during backpropagation.
  4. 4. Which is a typical pattern for deep networks?

    It keeps compile time flat in depth and limits stored activations.

Summary

  • lax.scan(f, init, xs) compiles a loop once; the carry must keep a fixed shape and dtype.
  • Python loops are unrolled under jit; use scan for long loops and recurrences.
  • Backprop stores forward activations; memory grows with depth, batch and sequence length.
  • jax.checkpoint recomputes them on demand: less memory, about one extra forward pass.
  • Combine scan and remat for deep or long models; use policies to choose what to keep.

Next: Mixed Precision, Buffer Donation and Debugging