Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 40 ยท Parallelism and Performance

Data Parallelism with pmap and Sharding

Train on many devices with JAX: split a batch across devices, average gradients with pmean, then use the modern jax.sharding Mesh and NamedSharding approach with jit.

One accelerator is not always enough. Data parallelism is the simplest way to use several: every device holds a full copy of the model, each processes a different slice of the batch, and the devices average their gradients so all copies stay identical. JAX supports this with two APIs. pmap is the original, explicit one. The newer sharding system lets you annotate arrays with how they are laid out and have jit do the rest.

Multi-device code needs real devices (several GPUs or TPU cores) or CPU devices simulated with a flag, and JAX is not available in this site's live editors. The JAX snippets are read-only. The runnable blocks simulate the same arithmetic with NumPy lists standing in for devices, so you can verify the key claim: averaging per-device gradients equals the full-batch gradient.

The idea in one picture

flowchart LR
    B["global batch (64)"] --> S["split into 4 shards of 16"]
    S --> D0["device 0<br/>loss + grad"]
    S --> D1["device 1<br/>loss + grad"]
    S --> D2["device 2<br/>loss + grad"]
    S --> D3["device 3<br/>loss + grad"]
    D0 & D1 & D2 & D3 --> AR["all-reduce: mean of grads"]
    AR --> U["same update on every device"]

Parameters are replicated; only the data is split. Communication happens once per step, for the gradient average.

Simulating it: the math checks out

When every shard has the same size and the loss is a mean, the mean of the shard gradients equals the gradient of the mean over the whole batch. The block below uses a linear model with a hand-derived gradient.

Python Mean of per-device gradients equals the full-batch gradient (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If shard sizes are unequal, a plain mean of shard means is not the global mean. Keep shards equal, or weight by shard size.

pmap: explicit device parallelism

jax.pmap compiles a function and runs it on every device at once. The first axis of each argument is the device axis: its size must not exceed the device count. Inside, you use collective operations such as jax.lax.pmean along a named axis.

Python
import jax, jax.numpy as jnp
from functools import partial

n = jax.local_device_count()

def loss_fn(params, x, y):
    return jnp.mean((x @ params - y) ** 2)

@partial(jax.pmap, axis_name="batch")
def train_step(params, x, y):
    loss, grads = jax.value_and_grad(loss_fn)(params, x, y)
    grads = jax.lax.pmean(grads, axis_name="batch")     # average across devices
    loss = jax.lax.pmean(loss, axis_name="batch")
    return params - 0.1 * grads, loss

params = jnp.zeros(3)
replicated = jax.device_put_replicated(params, jax.local_devices())  # shape (n, 3)
xb = X.reshape(n, -1, 3)       # (n_devices, per_device_batch, features)
yb = y.reshape(n, -1)
replicated, loss = train_step(replicated, xb, yb)

Points to remember:

  • Replicate parameters (device_put_replicated) and shard the batch (reshape so the leading axis is the device count).
  • Results come back with a leading device axis too; take [0] to read one copy.
  • Random keys must differ per device if you want different dropout masks: split a key into n keys and pass one to each (see PRNG keys).
  • pmap is simple and still works, but for new code the sharding API is recommended, and it scales better to model parallelism.

Sharding: describe the layout, let jit do the work

With the sharding API you create a mesh of devices with named axes, then describe how each array is laid out with a PartitionSpec. A jit-compiled function over sharded inputs is automatically partitioned (by the XLA SPMD compiler): it inserts the communication for you.

Python
from jax.sharding import Mesh, NamedSharding, PartitionSpec as P
import numpy as np

devices = np.array(jax.devices())
mesh = Mesh(devices, axis_names=("data",))

batch_sharding = NamedSharding(mesh, P("data"))      # split axis 0 across "data"
param_sharding = NamedSharding(mesh, P())            # empty spec = replicate

x = jax.device_put(x_global, batch_sharding)
y = jax.device_put(y_global, batch_sharding)
params = jax.device_put(params, param_sharding)

@jax.jit
def train_step(params, x, y):
    loss, grads = jax.value_and_grad(loss_fn)(params, x, y)
    return params - 0.1 * grads, loss

params, loss = train_step(params, x, y)       # no pmean written anywhere

Notice what is missing: no pmean, no device axis in the shapes. The loss is a mean over the whole (logically global) batch, and the compiler works out that this needs an all-reduce. The code reads like single-device code, and the layout lives in the shardings. You can print x.sharding or call jax.debug.visualize_array_sharding(x) to inspect how an array is split.

Meshes with two axes

The same mechanism supports model parallelism. A 2-D mesh ("data", "model") can split the batch along data and a large weight matrix along model:

Python
mesh = Mesh(devices.reshape(4, 2), axis_names=("data", "model"))
x_sharding = NamedSharding(mesh, P("data", None))     # rows by data
w_sharding = NamedSharding(mesh, P(None, "model"))    # columns by model

Here the compiler inserts whatever gathers and reductions the matrix multiplication needs. None in a PartitionSpec means "not split along that dimension".

Python Which slice of the array lives on which device? (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Practical advice

  • Test on one device first. Correct single-device code with fixed shapes is the base for everything else.
  • Simulate devices on a CPU to debug: set XLA_FLAGS="--xla_force_host_platform_device_count=8" before importing JAX, then jax.devices() lists eight CPU devices.
  • Per-device batch size should be the global batch divided by the device count; the global batch must divide evenly.
  • BatchNorm statistics are per device under pmap unless you pass an axis_name to average them.
  • Multi-host training (several machines) uses jax.distributed.initialize(); each process sees its own local devices, and the same mesh code scales out. See async dispatch and benchmarking before timing any of this.
  • Communication costs time. Gradient averaging scales with parameter size, so small models on many devices can be slower than on one.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In data parallelism, what is replicated and what is split?

    Each device holds a full copy of the parameters and processes its own slice of the batch.
  2. 2. What does jax.lax.pmean(grads, 'batch') do inside pmap?

    pmean is an all-reduce average over the named mapped axis.
  3. 3. In the sharding API, what does PartitionSpec('data', None) mean for a 2-D array?

    Each entry corresponds to an array dimension; None means that dimension is not partitioned.
  4. 4. Why can the sharded jit train step omit an explicit pmean?

    XLA's SPMD partitioner derives the communication from the shardings.

Summary

  • Data parallelism replicates parameters, splits the batch and averages gradients.
  • pmap with lax.pmean is explicit; the device axis comes first in every argument.
  • With Mesh, NamedSharding and PartitionSpec, plain jit is partitioned automatically and the code looks single-device.
  • A 2-D mesh adds model parallelism by splitting weights along another axis.
  • Keep shards equal and debug on simulated CPU devices before using real hardware.

Next: lax.scan, Rematerialization and Memory