Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 45 ยท Saving, Speed and Hardware

Distributed Training Overview

Understand PyTorch multi-GPU training: DataParallel versus DistributedDataParallel, process groups, gradient synchronization, FSDP and torchrun, with a NumPy simulation.

One GPU eventually becomes too slow, too small, or both. Distributed training spreads the work over several GPUs, on one machine or on many. There are two different reasons to do it, and they need different tools: to train faster on more data per second (the model still fits on one GPU), or to train a model that is too large for any single GPU's memory.

PyTorch cannot run on this site, and there is only one browser process here, so the PyTorch code is read-only. The live editors use NumPy to simulate the core idea of data parallelism: several "workers" compute gradients on different slices of a batch, the gradients are averaged, and everyone applies the same update. That simulation is a stand-in for what DistributedDataParallel does across real processes.

Two reasons to go distributed

Speed Same model, more data per second.
Size A model too big for one GPU.

The first is data parallelism, the second needs sharding.

1 / 2

DataParallel versus DistributedDataParallel

PyTorch has two data-parallel wrappers, and only one of them is the right choice today.

nn.DataParallel is a one-line change (model = nn.DataParallel(model)) that runs in a single process using threads. Each step it scatters the batch to every GPU, replicates the model, gathers the outputs back on GPU 0 and computes the loss there. That is convenient, but GPU 0 does extra work and holds extra memory, and Python's global interpreter lock limits scaling. It is effectively legacy.

DistributedDataParallel (DDP) uses one process per GPU. Each process owns its GPU, builds the model once, and processes its own batches. Processes only talk to each other to synchronize gradients, using fast GPU-to-GPU communication. It scales well, works across machines, and is the standard way to train on multiple GPUs. Use it even for two GPUs on one machine.

Processes, ranks and the process group

In DDP, every process has a number. The world size is the total number of processes, the rank is a process's global index (0 to world size minus 1), and the local rank is its GPU index on its own machine. Processes find each other through a process group and communicate using a backend: nccl for NVIDIA GPUs, gloo for CPUs (useful for testing on a laptop).

Python
import os
import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP

def main():
    dist.init_process_group(backend="nccl")        # reads env vars set by torchrun
    local_rank = int(os.environ["LOCAL_RANK"])
    torch.cuda.set_device(local_rank)
    device = torch.device("cuda", local_rank)

    model = MyModel().to(device)
    model = DDP(model, device_ids=[local_rank])    # wraps the model

    optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4)

    for xb, yb in loader:                          # a sharded loader, see below
        xb, yb = xb.to(device), yb.to(device)
        loss = loss_fn(model(xb), yb)
        optimizer.zero_grad(set_to_none=True)
        loss.backward()                            # gradients are averaged here
        optimizer.step()

    dist.destroy_process_group()

if __name__ == "__main__":
    main()

Notice how little changes compared with a single-GPU loop: wrapping the model in DDP is the main difference. The wrapper registers hooks on every parameter. As backward() computes gradients layer by layer, DDP starts an all-reduce on groups ("buckets") of finished gradients while the rest of the backward pass is still running, so communication overlaps with computation. When backward() returns, every process holds the same averaged gradients, so every optimizer takes the same step and the model copies stay identical.

Launching with torchrun

You do not start the processes yourself. torchrun launches one copy of your script per GPU and sets the environment variables (RANK, LOCAL_RANK, WORLD_SIZE, MASTER_ADDR, MASTER_PORT) that init_process_group reads.

Terminal
# one machine, 4 GPUs
torchrun --standalone --nproc_per_node=4 train.py

# two machines, 8 GPUs each; run on each machine with its own node_rank
torchrun --nnodes=2 --nproc_per_node=8 --node_rank=0 \
         --rdzv_backend=c10d --rdzv_endpoint=host0:29500 train.py

Giving each process different data

If all four processes read the same batches, you have wasted three quarters of the work. The DistributedSampler gives each rank a distinct shard of the dataset:

Python
from torch.utils.data import DataLoader, DistributedSampler

sampler = DistributedSampler(dataset, shuffle=True)   # uses rank and world size
loader = DataLoader(dataset, batch_size=64, sampler=sampler,
                    num_workers=4, pin_memory=True)

for epoch in range(num_epochs):
    sampler.set_epoch(epoch)      # reshuffle differently every epoch
    for xb, yb in loader:
        ...

Forgetting set_epoch means every epoch uses the same shuffle order. Also remember what the numbers mean: batch_size=64 is per process, so with 4 GPUs the effective batch is 256. Many people scale the learning rate with the effective batch size and add a warm-up period (see Learning Rate Schedules).

Everything that should happen once (printing, saving checkpoints, writing logs) must be guarded so it runs only on rank 0:

Python
if dist.get_rank() == 0:
    torch.save(model.module.state_dict(), "checkpoint.pt")   # .module unwraps DDP

Saving model.module.state_dict() stores clean parameter names without the module. prefix. See Saving and Loading Models.

What synchronization actually does

The operation DDP relies on is all-reduce: every process contributes a tensor and every process receives the sum (or average) of all contributions. The simulation below plays the role of four workers on one batch. Each computes the gradient of its slice, the gradients are averaged, and the result is compared with the gradient of the whole batch computed by one process.

Python Simulate four data-parallel workers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The averaged gradient equals the full-batch gradient when the shards have equal size, which is why DDP with 4 GPUs and per-GPU batch size 64 behaves like one GPU with batch 256. Because every replica applies the same gradient to the same starting weights, they stay bit-for-bit identical, with no need to ever copy weights between processes.

How an all-reduce is built

Sending all gradients to one process and back would make that process a bottleneck. NCCL uses a ring all-reduce: the processes form a ring, split the tensor into pieces, and pass pieces around so that every link carries data at the same time. Each process sends and receives roughly 2 * (n - 1) / n times the gradient size in total, which barely grows as you add GPUs, so communication cost per GPU stays almost flat. The editor simulates the idea with plain lists.

Python A ring all-reduce simulated

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each worker sends about twice the size of the gradients no matter how many workers there are. That is why data parallelism scales well until the network, not the GPU, becomes the limit.

When the model does not fit: FSDP

DDP keeps a full copy of parameters, gradients and optimizer state on every GPU, as described in GPUs, Devices and Mixed Precision. For very large models that is the bottleneck. Fully Sharded Data Parallel (FSDP) keeps only a shard of everything on each GPU. Just before a layer runs, the GPUs gather that layer's full parameters from each other (an all-gather), compute, and immediately free the borrowed pieces; during backward, gradients are reduce-scattered so each GPU keeps only the part matching its shard.

Python
from torch.distributed.fsdp import fully_shard

for block in model.blocks:      # shard each transformer block separately
    fully_shard(block)
fully_shard(model)              # then the root module

With 8 GPUs, each holds roughly one eighth of the weights, gradients and optimizer state, at the cost of more communication. A rough memory sketch with Adam: 16 bytes per parameter becomes 16/8 = 2 bytes per parameter per GPU for states that are sharded (activations are not reduced by sharding the parameters). The FSDP API has changed between releases (FullyShardedDataParallel wrapper versus fully_shard), so use the version that matches your installation. Beyond FSDP there are also tensor parallelism (splitting individual matrices over GPUs) and pipeline parallelism (assigning layers to different GPUs), which large language model training combines.

Common mistakes

  • Using nn.DataParallel for new code instead of DDP.
  • Forgetting DistributedSampler, so every GPU trains on the same data.
  • Forgetting sampler.set_epoch(epoch), so shuffling repeats.
  • Saving or logging from every rank, which overwrites files and floods output; guard with rank == 0.
  • Misreading the batch size: it is per process, the effective batch is batch_size * world_size.
  • Different processes taking different code paths around backward(), which can hang a collective operation.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How does DistributedDataParallel keep the model copies identical on every GPU?

    All-reduce averages gradients during backward. Same starting weights plus the same gradient gives the same new weights on every rank.
  2. 2. With torchrun on 4 GPUs and DataLoader batch_size=32, what is the effective batch size per optimizer step?

    The batch size is per process. Four processes each handle 32 samples, so one step sees 128 samples.
  3. 3. What is DistributedSampler for?

    Without it every process would iterate over the entire dataset, repeating the same work on every GPU.
  4. 4. What is the main idea of FSDP compared with DDP?

    Sharding cuts per-GPU memory roughly by the number of GPUs for the sharded states, at the cost of extra communication.

Summary

  • Use DistributedDataParallel, one process per GPU, rather than nn.DataParallel.
  • Each process has a rank and a local rank; torchrun launches the processes and sets the environment variables.
  • DDP averages gradients with an all-reduce during backward(), so all replicas apply identical updates.
  • Use DistributedSampler (plus set_epoch) and remember the batch size is per process.
  • Do logging and checkpointing on rank 0 only; save model.module.state_dict().
  • FSDP shards parameters, gradients and optimizer state to train models that do not fit on one GPU.

Next, learn to find the mistakes that all of this machinery makes easy in Debugging PyTorch Code.