Skip to content
Tutorials Center
My learning

Topics

JAX Tutorial

Learn JAX step by step: 40 lessons in 9 chapters with examples, quizzes, diagrams and projects.

Start learning →

of lessons completed

Getting Started

  1. 1 What Is JAX? Meet JAX: NumPy-style arrays plus composable function transformations (grad, jit, vmap) that run on CPU, GPU and TPU through the XLA compiler.
  2. 2 Installing JAX and Choosing Devices Install JAX for CPU, NVIDIA GPU or TPU with pip, verify the installation, list devices, understand platforms and backends, and fix common setup problems.
  3. 3 JAX vs NumPy Compare jax.numpy with NumPy: shared API, immutability, float32 defaults, asynchronous dispatch, random numbers, and when to convert between the two libraries.
  4. 4 Functional Programming in JAX Learn why JAX needs pure functions, what side effects break, how to pass state explicitly, and how this style makes grad, jit and vmap safe to combine.

Arrays and NumPy API

  1. 5 JAX Arrays and Data Types Create and inspect JAX arrays, understand dtypes, float32 defaults, bfloat16, weak typing and type promotion, and convert between JAX, NumPy and Python values.
  2. 6 Immutability and .at Updates Why JAX arrays cannot be modified in place, how .at[idx].set/add/mul/min/max return updated copies, out-of-bounds modes, and how jit makes updates efficient.
  3. 7 Indexing, Slicing and Broadcasting Master JAX array indexing: slices, negative indexes, fancy and boolean indexing, jnp.take, clamped reads, and NumPy broadcasting rules with practical examples.
  4. 8 Reductions, Linear Algebra and Math Functions Use jax.numpy for reductions along axes, cumulative ops, sorting, matrix products, jnp.linalg solves and decompositions, einsum, and jax.scipy special functions.
  5. 9 Low-level Operations and Control Flow with lax Learn jax.lax, the low-level API behind jax.numpy: lax.cond, switch, fori_loop, while_loop and scan, with pure-Python stand-ins that show exactly what each does.

Autodiff with grad

  1. 10 grad Basics: Differentiating Python Functions Learn jax.grad: differentiate ordinary Python functions, use loops and if statements, handle vectors, understand scalar outputs and float inputs, and check gradients numerically.
  2. 11 value_and_grad, argnums and Auxiliary Data Get the loss and its gradient in one pass with jax.value_and_grad, differentiate with respect to several arguments using argnums, and return extra metrics with has_aux.
  3. 12 Jacobians, Hessians and Higher-Order Derivatives Differentiate vector-valued functions with jax.jacfwd and jax.jacrev, build Hessians, and understand when forward or reverse mode is cheaper. Includes runnable stand-ins.
  4. 13 stop_gradient and Custom Derivatives Control autodiff in JAX: block gradients with stop_gradient, fix numerically unstable derivatives with custom_jvp, and define your own backward pass with custom_vjp.
  5. 14 Forward and Reverse Mode: jvp and vjp Understand the two building blocks of JAX autodiff: jax.jvp for forward mode and jax.vjp for reverse mode, how they relate to grad and Jacobians, and when each is cheaper.

jit and XLA

  1. 15 jit Basics: Compiling Functions with XLA Learn what jax.jit does, how XLA fuses operations into fast kernels, how to apply it as a decorator, and when compilation pays off. With runnable stand-ins that show the idea.
  2. 16 Tracing and Static Arguments Understand how jax.jit traces functions with abstract values, why shapes must be known, what a tracer is, and when to use static_argnums or static_argnames.
  3. 17 jit with Control Flow and Recompilation Write branches and loops that compile under jax.jit: jnp.where, lax.cond, lax.scan and fori_loop, plus how to spot and avoid accidental recompilation and padding tricks.
  4. 18 jit Pitfalls and Side Effects Avoid the classic jax.jit bugs: prints that run once, mutated globals, captured constants, impure functions and in-place updates. Learn jax.debug.print and checkify.
  5. 19 Async Dispatch and Benchmarking JAX Code Time JAX code the right way: understand asynchronous dispatch, use block_until_ready, separate compile time from run time, and avoid common benchmarking mistakes.
  6. 20 Inspecting jaxprs, StableHLO and Compiled Code Look inside JAX: read jaxprs with make_jaxpr, view StableHLO and optimized HLO with lower and compile, check cost analysis, and understand what XLA did with your function.

vmap and Vectorization

  1. 21 vmap Basics: Write One Example, Run Many Learn what jax.vmap does, how it turns a function for one example into a batched one, and how it compares with Python loops and manual broadcasting.
  2. 22 vmap Axes, Nesting and Pairwise Computations Control which axis vmap maps with in_axes and out_axes, nest vmaps to build pairwise distance matrices, and know when to use jnp.vectorize or lax.map instead.
  3. 23 Per-Example Gradients with vmap and grad Combine jax.grad with jax.vmap to get one gradient per training example, then use them for gradient clipping, noise estimates and influence scores.
  4. 24 vmap with Conditionals and Its Limits Understand why Python if fails under vmap, how lax.cond becomes select, why where can produce NaN gradients, and what batched while_loop really costs.

Random Numbers and Pytrees

  1. 25 PRNG Keys and split: Random Numbers the JAX Way Learn why JAX uses explicit PRNG keys instead of a global seed, how jax.random.split and fold_in work, and the one rule that prevents correlated random numbers.
  2. 26 Random Sampling Patterns: Distributions, Shuffles and Dropout Use jax.random distributions, shuffle data with permutation, draw weighted choices, build dropout masks and run Monte Carlo batches with one key per run.
  3. 27 Pytrees Basics: Nested Data JAX Understands Understand JAX pytrees: nested lists, tuples and dicts of arrays. Learn tree_map, leaves, structure, flatten and unflatten, tree reduce and common pitfalls.
  4. 28 Custom Pytrees: Registering Your Own Classes Make your own classes work with jit, grad and vmap: NamedTuple, register_dataclass, register_pytree_node, static fields, and the flatten and unflatten contract.

Neural Networks with Flax

  1. 29 Flax Modules: init and apply Learn how Flax separates a model from its parameters: define layers with nn.Module, create parameters with init, run them with apply, and inspect the parameter tree.
  2. 30 Optax and TrainState: Optimizers as Pure Functions Understand how Optax optimizers work as pure init and update functions, how to chain clipping and weight decay, add learning-rate schedules, and use Flax TrainState.
  3. 31 Training an MLP: The Full Flax Loop Build a complete JAX training loop: loss function, jitted train step, minibatches with epoch shuffling, evaluation, and the mistakes that make loops slow or wrong.
  4. 32 Dropout, BatchNorm and the NNX API Handle layers that behave differently in training and evaluation in Flax: dropout RNG keys, BatchNorm running statistics, mutable collections, and a first look at NNX.
  5. 33 Convolutional Networks in Flax Build a CNN with Flax: nn.Conv, NHWC layout, pooling, flattening, parameter shapes, and how lax.conv works underneath, with a runnable NumPy convolution you can inspect.
  6. 34 Saving Models, Learning-Rate Schedules and Gradient Clipping Save and restore JAX parameter pytrees, add warmup and cosine learning-rate schedules with Optax, clip gradients by global norm, and chain optimizer transformations.

Parallelism and Performance

  1. 35 Data Parallelism with pmap and Sharding Train on many devices with JAX: split a batch across devices, average gradients with pmean, then use the modern jax.sharding Mesh and NamedSharding approach with jit.
  2. 36 lax.scan, Rematerialization and Memory Compile long loops with lax.scan, trade compute for memory with jax.checkpoint (remat), and understand why backprop stores activations, with runnable NumPy stand-ins.
  3. 37 Mixed Precision, Buffer Donation and Debugging Speed up JAX programs with bfloat16 and float16, reuse memory with donate_argnums, and debug jitted code with jax.debug.print, NaN checks and disabling jit.

Projects and Practice

  1. 38 Project: Fit a Curve with Gradient Descent A complete JAX project: define a model and loss, take gradients with jax.grad, jit a train step, fit noisy data with a pytree of parameters, and check gradients numerically.
  2. 39 Project: Simulate a Pendulum with scan and grad Write a differentiable physics simulation in JAX: integrate a pendulum with lax.scan, then differentiate through the whole trajectory to estimate an unknown parameter.
  3. 40 Practice Problems and a Troubleshooting Guide Test your JAX skills with graded exercises on grad, vmap, jit, scan and pytrees, plus a guide to the most common JAX errors, their causes and their fixes.