Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 40 ยท Projects and Practice

Practice Problems and a Troubleshooting Guide

Test your JAX skills with graded exercises on grad, vmap, jit, scan and pytrees, plus a guide to the most common JAX errors, their causes and their fixes.

This last lesson has two parts. First, a set of practice problems in the order of the course, each with a hint and a runnable NumPy check you can adapt (JAX does not run in this site's editors, so the checks use NumPy and plain Python to verify the ideas). Second, a field guide to the errors you will meet most often, with the cause and the fix for each.

Try each problem on your own machine with real JAX before looking at the hints.

Practice problems

1. Arrays and updates

Write normalize_rows(x) that divides each row of a matrix by its L2 norm without any Python loop, then a function that sets the diagonal of a square matrix to zero using .at[...].set(...).

Hint: jnp.linalg.norm(x, axis=1, keepdims=True); indices from jnp.arange(n); see immutability and at updates.

Python Check for problem 1 (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

2. Gradients

Write f(x) = sum(sin(x) * x**2) and compute its gradient with jax.grad. Compare with the formula cos(x) * x**2 + 2 * x * sin(x). Then compute the Hessian diagonal. Hints: grad basics, Jacobians and Hessians.

Python Check for problem 2: gradient by formula versus finite differences

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

3. vmap

Given a function dist(a, b) for two vectors, produce the full pairwise distance matrix for two sets of points with nested vmap and no loops. Hint: vmap(vmap(dist, (None, 0)), (0, None)); see vmap axes and nesting.

Python Check for problem 3: pairwise distances with broadcasting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

4. jit and static arguments

A function power(x, n) builds x ** n with a Python for loop over range(n). Why does jitting it with n as a normal argument fail, and how do you fix it? Hint: static_argnums; see tracing and static arguments.

5. Random numbers

Draw 1,000 samples from a normal distribution with a split key; show that using the same key twice gives identical samples. Then derive 5 independent keys for 5 workers. Hint: PRNG keys and split.

Python Check for problem 5: keys as seeds (NumPy stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

6. Pytrees

Write a function that takes a nested dict of parameters and returns the total number of scalar parameters and the global L2 norm. Hint: jax.tree_util.tree_leaves; see pytrees basics.

7. scan

Compute the running maximum of an array with lax.scan. Then compute the Fibonacci numbers with scan and a carry of two numbers. Hint: scan, remat and memory.

Python Check for problem 7: running max and Fibonacci as scans

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

8. Capstone

Combine the lessons: fit a two-layer network to y = sin(x) with a dict of parameters, a jitted value_and_grad step, per-epoch fold_in shuffling and an Optax Adam optimizer. Add gradient clipping and a cosine schedule from the checkpoints lesson, then save the final parameters.

Troubleshooting guide

Error or symptom Usual cause Fix
TracerBoolConversionError / ConcretizationTypeError A Python if, while or range(n) uses a traced value jnp.where, lax.cond, lax.scan, or mark the value static (control flow under jit)
TypeError: ... does not support item assignment Writing x[i] = v on a JAX array x = x.at[i].set(v)
UnexpectedTracerError A traced value escaped a transformed function (stored in a global or list) Return values instead of stashing them (pitfalls)
Recompiles on every call Shapes or static arguments change Pad or bucket shapes; hash-stable static args
NaNs after a few steps log(0), bad learning rate, where trap jax_debug_nans, clip gradients, double-where (debugging)
Same random numbers every call Reusing one key split a new key each time
TypeError: Gradient only defined for scalar-output functions grad of a function returning an array Reduce to a scalar, or use jacobian / vjp
Output differs after jit A side effect or a Python-level value baked in at trace time Pass it as an argument; use jax.debug.print for output
Slow timings Measuring dispatch not computation block_until_ready(), warm up first (benchmarking)
Shapes must be 1D sequences of concrete values of integer type A traced array used as a shape Make that value static
Out of memory Large activations, big batch Smaller batch, jax.checkpoint, mixed precision, sharding
float64 requested, float32 given x64 mode is off jax.config.update("jax_enable_x64", True) at program start

A habit that prevents most problems

Write the function for one example with fixed shapes, test it without jit, then add vmap, grad, and jit in that order, checking shapes after each. When something breaks, jax.make_jaxpr(f)(x) and jax.eval_shape(f, x) show what JAX sees without running the computation.

Where to go next

  • Read the Flax and Optax documentation for current APIs: they evolve faster than the core library.
  • Try the real code from this course on a free GPU notebook service and measure speedups with correct benchmarking.
  • Explore the ecosystem: Equinox (modules as pytrees), Diffrax (differential equations), BlackJAX (sampling) and jaxopt (optimizers).
  • Re-implement a classic model (a small transformer, a variational autoencoder) with only jax.numpy, then with Flax.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. You get a TracerBoolConversionError. What is the likeliest cause?

    During tracing values are abstract, so Python cannot decide a branch; use jnp.where or lax.cond.
  2. 2. Your jitted function is slow because it recompiles each call. What should you check first?

    jit caches by shapes, dtypes and static arguments; changes trigger new compilations.
  3. 3. What is the recommended order for building up a JAX function?

    Each transformation adds a layer of complexity; debug simple versions first.
  4. 4. How do you get float64 values in JAX?

    JAX defaults to 32-bit; the flag must be set before arrays are created.

Summary

  • Practice each tool in isolation (arrays, grad, vmap, jit, keys, pytrees, scan), then combine them in a capstone.
  • Check gradients and transformations against simple NumPy or loop versions.
  • Most errors come from Python control flow on traced values, in-place updates, key reuse or changing shapes.
  • Build functions for one example first, then add transformations in order.
  • Use make_jaxpr, eval_shape, jax.debug.print and jax_debug_nans to see what is happening.

Next: Back to the JAX track page