Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 45 ยท Saving, Speed and Hardware

Making Training Faster

Find and fix PyTorch bottlenecks: profiling, DataLoader workers, torch.compile, channels_last, gradient accumulation and benchmarking, with live NumPy demos.

Before you make training faster, you have to know what is slow. Experienced engineers follow one rule above all others: measure first. A model can be slow because the math is heavy, because the GPU is waiting for data, because Python is launching thousands of tiny operations, or because memory is moving more than it needs to. Each cause has a different cure, and guessing wrong wastes days.

This lesson gives you a checklist from cheapest fix to most advanced. The PyTorch code is read-only, since PyTorch cannot run on this site. The live editors use NumPy to demonstrate two principles that hold in every numerical library: doing work in big vectorized chunks beats Python loops, and gradient accumulation gives the same update as one big batch. The timing numbers you see will vary from run to run, which is itself a lesson about benchmarking.

Where does time go?

Data loading The GPU waits for the next batch.
Python overhead Thousands of tiny operations.
Compute Big matrix math. Use fast dtypes.
Memory traffic Reading and writing tensors.

Profile first, then fix the biggest slice.

1 / 2

Measure before you optimize

Time a piece of code with torch.utils.benchmark, which handles warm-up and, on GPUs, synchronization for you. Plain time.time() is misleading for GPU code, because kernels run asynchronously and the clock stops before the work finishes.

Python
import torch
from torch.utils import benchmark

x = torch.randn(1024, 1024)

timer = benchmark.Timer(
    stmt="x @ x",
    globals={"x": x},
)
print(timer.timeit(50))        # mean time per call, with spread

To find out which operations dominate a whole training step, use the profiler:

Python
from torch.profiler import profile, ProfilerActivity

with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
             record_shapes=True) as prof:
    for _ in range(5):
        loss = loss_fn(model(xb), yb)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))

The table lists the most expensive operators. If the GPU column is mostly idle while the CPU column is busy, you are bound by Python or data loading, not by arithmetic. A quick sanity check for the data pipeline: time a loop over the DataLoader that does nothing with the batches. If that alone takes as long as an epoch, the model is not your bottleneck.

Fix the input pipeline first

A fast GPU fed by a slow DataLoader spends most of its life idle. The usual levers:

Python
loader = DataLoader(
    dataset,
    batch_size=256,
    shuffle=True,
    num_workers=8,            # decode/augment in parallel worker processes
    pin_memory=True,          # faster host-to-GPU copies
    persistent_workers=True,  # keep workers alive between epochs
    prefetch_factor=4,        # batches prepared ahead per worker
)

num_workers=0 (the default) loads data in the main process, one batch at a time, while the GPU waits. A reasonable starting point is the number of CPU cores you can spare; more workers than cores only adds overhead. Move expensive per-sample work (image decoding, resizing) into the workers, or do it once and cache the result on disk. See GPUs, Devices and Mixed Precision for pin_memory and non_blocking.

Use the hardware's fast paths

Several cheap switches change how the same model runs:

Python
# Mixed precision: autocast + bfloat16 (or float16 with GradScaler)
with torch.autocast("cuda", dtype=torch.bfloat16):
    loss = loss_fn(model(xb), yb)

# Faster float32 matmuls on recent GPUs (small precision loss)
torch.set_float32_matmul_precision("high")

# Let cuDNN pick the fastest convolution algorithm for fixed input sizes
torch.backends.cudnn.benchmark = True

# Convolution-friendly memory layout on modern GPUs
model = model.to(memory_format=torch.channels_last)
xb = xb.to(memory_format=torch.channels_last)

# Cheaper zeroing of gradients
optimizer.zero_grad(set_to_none=True)

channels_last stores image tensors as (batch, height, width, channels) in memory while keeping the logical shape (batch, channels, height, width). Convolutions on Tensor Core GPUs are usually faster in that layout, and the effect is mostly about memory access, not about what the numbers mean. The cudnn.benchmark flag helps when input sizes never change; if they vary every batch it keeps re-tuning and slows you down.

torch.compile

Eager PyTorch runs one Python-level operation at a time: each addition, multiplication and activation launches its own GPU kernel and writes its result to memory. torch.compile captures your model's operations as a graph and generates fused kernels, so a chain like "add, multiply, ReLU" becomes one kernel with one memory read and one write.

Python
compiled = torch.compile(model)           # returns a wrapper, same interface

for xb, yb in loader:
    loss = loss_fn(compiled(xb), yb)      # first call is slow: it compiles
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

What to expect: the first iterations are slow (compilation can take from seconds to minutes), later ones are faster, typically somewhere between no gain and about twice as fast depending on the model. Models made of many small elementwise operations benefit most; a model dominated by a few huge matrix multiplications already runs near the hardware limit. The compiler recompiles when input shapes change, so variable-length batches can trigger repeated compilation; mark the dimension as dynamic (torch.compile(model, dynamic=True)) or pad to a few fixed sizes. Python control flow that depends on tensor values causes a "graph break", where the compiler falls back to eager mode for that part. torch._dynamo.explain(model)(x) shows where breaks happen. The details of the compiler change with each PyTorch release, so check the current notes for your version.

Python overhead and why vectorizing wins

The idea behind fused kernels and large batches is the same as behind NumPy's speed: each Python-level call costs a fixed amount, and a big array operation pays that cost once. A loop that calls a tiny operation a million times pays it a million times. Here is the effect, measured in your own browser:

Python Python loop versus one vectorized operation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The exact milliseconds depend on your machine, but the gap is large. Best-of-three timing, with the minimum rather than the average, is the standard way to reduce noise from other programs running at the same time: the minimum is the run that was disturbed least. The same trick, applied to the GPU, is why benchmark.Timer repeats the measurement.

Gradient accumulation

When a big batch does not fit in memory, process several small micro-batches, add up their gradients, and take one optimizer step. Because gradients accumulate in .grad (see Autograd in Practice), the code is short:

Python
accum_steps = 4                      # effective batch = 4 * micro-batch size
optimizer.zero_grad(set_to_none=True)

for i, (xb, yb) in enumerate(loader):
    loss = loss_fn(model(xb.to(device)), yb.to(device))
    (loss / accum_steps).backward()  # divide so the sum is an average

    if (i + 1) % accum_steps == 0:
        optimizer.step()
        optimizer.zero_grad(set_to_none=True)

The division matters: losses are averages over each micro-batch, so summing four of them gives four times the gradient of one big batch. One caution: layers whose behavior depends on the batch (BatchNorm) see only the micro-batch, so accumulation is not exactly equivalent for them. The editor proves the equivalence for a model without such layers.

Python Accumulating gradients equals one big batch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A practical order of attack

  1. Confirm the GPU is busy (nvidia-smi utilization). If it is not, fix the DataLoader.
  2. Use the largest batch that fits, with mixed precision.
  3. Turn on channels_last for CNNs and set_float32_matmul_precision("high").
  4. Try torch.compile and keep it only if the steady-state step time improves.
  5. Use gradient accumulation when memory, not speed, is the limit.
  6. Scale to several GPUs (see Distributed Training Overview).

Change one thing at a time and re-measure; speed-ups interact, and an optimization that helps one model can hurt another.

Common mistakes

  • Timing GPU code without synchronizing, which measures only the launch.
  • Including the first (compiling or warming-up) iterations in a benchmark.
  • Raising num_workers far above the CPU core count.
  • Forgetting to divide the loss by accum_steps, which silently multiplies the learning rate.
  • Calling .item() or .cpu() on a tensor inside every step. It forces the GPU to finish before Python continues. Log every N steps instead.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Your GPU utilization is low and an epoch takes as long with an empty training step as with the real one. Where is the bottleneck?

    If looping over the DataLoader alone takes the whole epoch time, the model is waiting for data. Increase workers, cache decoded data and use pin_memory.
  2. 2. What does torch.compile mainly do to speed up a model?

    By capturing a graph, the compiler generates fused kernels that avoid launching many small ones and writing every intermediate result to memory.
  3. 3. With gradient accumulation over 4 micro-batches, why do you divide each loss by 4?

    Each micro-batch loss is already a mean. Summing four of them without dividing gives a gradient four times larger than one big batch.
  4. 4. Why is the minimum of several timing runs a good benchmark number?

    Noise from other programs adds time but never subtracts it, so the best run is closest to the true cost. Warm-up is still needed.

Summary

  • Measure first: torch.utils.benchmark for snippets, torch.profiler for whole steps, and always synchronize on GPUs.
  • Fix the DataLoader (num_workers, pin_memory, persistent_workers) before touching the model.
  • Mixed precision, channels_last, cudnn.benchmark and TF32-style matmul settings are cheap speed-ups.
  • torch.compile fuses operations; expect a slow first step, recompilation on shape changes and gains that vary by model.
  • Gradient accumulation (divide the loss by the number of steps) simulates a larger batch when memory is short.

Next, go beyond one device in Distributed Training Overview.