Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 45 ยท Saving, Speed and Hardware

GPUs, Devices and Mixed Precision

Run PyTorch on CPU, CUDA or Apple GPUs: .to(device), pinned memory, autocast, GradScaler, bfloat16 and memory use, with live float16 precision demos.

A GPU does thousands of small arithmetic operations at once, which is exactly what matrix multiplications in neural networks need. Moving a model to a GPU can make training ten to a hundred times faster, but it adds a rule you have to follow everywhere: tensors that meet in one operation must live on the same device. The second big speed-up is mixed precision: doing most arithmetic with 16-bit numbers instead of 32-bit ones, which halves memory traffic and unlocks special hardware units.

PyTorch cannot run in this browser and the site has no GPU, so the PyTorch snippets are read-only. The live editors use NumPy's float16 to let you see, with real numbers, why 16-bit arithmetic is both fast and fragile, and how loss scaling rescues it. Those demos are a stand-in for the numerical behavior; they do not measure GPU speed.

Where can a tensor live?

cpu Always available, always works.
cuda NVIDIA GPUs (and AMD through ROCm builds).
mps Apple silicon GPUs.

Pick one device string and pass it around your code.

1 / 2

Choosing a device

Write device-agnostic code once, and let one variable decide where things run.

Python
import torch

if torch.cuda.is_available():
    device = torch.device("cuda")
elif torch.backends.mps.is_available():
    device = torch.device("mps")
else:
    device = torch.device("cpu")

print(device)                      # e.g. cuda
x = torch.randn(3, 4)
print(x.device)                    # cpu: tensors start on the CPU
x = x.to(device)                   # copy to the GPU (returns a new tensor)
print(x.device)                    # cuda:0

.to(device) on a tensor returns a copy and does not change the original, so you must assign the result. On an nn.Module it moves the parameters in place and returns the module itself, so model.to(device) is enough. Create tensors directly on the target device when you can (torch.zeros(3, device=device)) to skip a copy.

Mixing devices is the most common GPU error:

Python
a = torch.randn(3, device="cuda")
b = torch.randn(3)                 # on the CPU
a + b
# RuntimeError: Expected all tensors to be on the same device,
# but found at least two devices, cuda:0 and cpu!

The fix is always the same: move inputs, labels and the model to the same device. In a training loop, that means moving each batch inside the loop:

Python
model = model.to(device)
for xb, yb in loader:
    xb, yb = xb.to(device), yb.to(device)
    loss = loss_fn(model(xb), yb)
    ...

Two more device facts. NumPy cannot read GPU memory, so call .cpu() before .numpy() (and .detach() first for tracked tensors). And GPU work is asynchronous: Python queues kernels and returns immediately, so timing code needs torch.cuda.synchronize() before reading the clock, and an error may be reported at a later line than the one that caused it (see Debugging PyTorch Code).

Feeding the GPU: pinned memory

Copying a batch from CPU to GPU takes time, and a GPU that waits for data is wasted money. Two settings help. pin_memory=True in the DataLoader puts batches in page-locked host memory that the GPU can read directly, and non_blocking=True lets the copy overlap with computation instead of stalling Python.

Python
loader = DataLoader(dataset, batch_size=128, shuffle=True,
                    num_workers=4, pin_memory=True)

for xb, yb in loader:
    xb = xb.to(device, non_blocking=True)
    yb = yb.to(device, non_blocking=True)

non_blocking only helps when the source is pinned; otherwise it behaves like a normal copy. Both are small wins on a single GPU and big ones when the GPU is fast and your data pipeline is slow.

Why 16-bit numbers

A float32 takes 4 bytes, a float16 or bfloat16 takes 2. Half the memory means you fit bigger batches or models, and half the data to move per operation. Modern GPUs also have dedicated units (such as Tensor Cores) that multiply 16-bit matrices several times faster than 32-bit ones. The price is range and precision, summarized here:

Type Bits Largest value Decimal digits Note
float32 32 about 3.4e38 about 7 the default
float16 16 65,504 about 3 narrow range, needs loss scaling
bfloat16 16 about 3.4e38 about 2 same range as float32, coarse

float16 has more precision but overflows above 65,504 and underflows (rounds to zero) below roughly 6e-8. bfloat16 keeps the exponent of float32, so it almost never overflows or underflows, but it has fewer digits. Try the numbers yourself:

Python The limits of 16-bit floats

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Case 4 is why mixed precision is "mixed": additions and reductions must happen in float32, even when the multiplications use 16-bit inputs. That is exactly what autocast arranges for you.

autocast: let PyTorch choose per operation

torch.autocast is a context manager. Inside it, operations that are safe and fast in 16 bits (matrix multiplies, convolutions) run in 16 bits; operations that are sensitive (softmax, loss functions, normalizations, large sums) are kept in float32. The model weights themselves stay in float32 as the "master copy", and the optimizer updates that copy.

Python
with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
    logits = model(xb)
    loss = loss_fn(logits, yb)

print(logits.dtype)    # torch.bfloat16 (output of a linear layer)
print(loss.dtype)      # torch.float32 (the loss is computed in float32)

loss.backward()        # backward is outside the with-block
optimizer.step()

Wrap only the forward pass and the loss; call backward() outside. With bfloat16 that is all you need, because the range matches float32. On older GPUs without bfloat16 support, use torch.float16 and add a gradient scaler.

GradScaler: keeping small gradients alive

Gradients are often tiny, far below float16's smallest representable number, so they would round to zero (case 3 above). Loss scaling multiplies the loss by a large factor (say 65,536) before backward(), which shifts all gradients up by the same factor into the representable range; the gradients are divided back down before the optimizer uses them. GradScaler automates this: it skips a step if it finds inf or NaN in the gradients, lowers the factor, and slowly raises it again when things look stable.

Python
scaler = torch.amp.GradScaler("cuda")

for xb, yb in loader:
    xb, yb = xb.to(device), yb.to(device)
    optimizer.zero_grad(set_to_none=True)

    with torch.autocast(device_type="cuda", dtype=torch.float16):
        loss = loss_fn(model(xb), yb)

    scaler.scale(loss).backward()     # scaled loss, scaled gradients
    scaler.unscale_(optimizer)        # optional: unscale before clipping
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    scaler.step(optimizer)            # skips the step if gradients are inf/NaN
    scaler.update()                   # adjust the scale factor

Clipping with clip_grad_norm_ needs real gradient values, so unscale first. The next editor shows the problem and the cure on 4,096 tiny gradient values: stored directly in float16 almost all of them become zero, but if they are multiplied by a scale first and divided back afterwards, the total survives.

Python Loss scaling rescues tiny gradients

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without scaling, nearly every value is flushed to zero and the recovered total is wrong by a factor of ten. A scale of 1,024 or more brings the total back, and an absurd scale produces inf and then nan. GradScaler searches for a good value on its own: it starts high, halves on overflow, and doubles after a long streak of clean steps.

Memory: what fills a GPU

GPU memory (VRAM) is the usual limit. Four things consume it:

  • Parameters: 4 bytes each in float32.
  • Gradients: one more copy of the parameters.
  • Optimizer state: Adam keeps two extra values per parameter, so 8 more bytes.
  • Activations: the intermediate results saved for backward(); they grow with batch size and sequence length, and are often the biggest part.

Training with Adam therefore needs about 16 bytes per parameter before activations. Mixed precision shrinks activations (not the master weights), a smaller batch size reduces them directly, and gradient accumulation (see Making Training Faster) gives you a large effective batch from several small ones. You can inspect what is going on:

Python
print(torch.cuda.memory_allocated() / 1e9, "GB in use by tensors")
print(torch.cuda.max_memory_allocated() / 1e9, "GB peak")
torch.cuda.empty_cache()   # returns cached blocks to the driver; rarely needed

If you hit CUDA out of memory, reduce the batch size first, then try mixed precision, and make sure you are not keeping loss tensors alive in a list (use loss.item(); see Autograd in Practice).

Common mistakes

  • Calling x.to(device) without assigning the result. Tensors are not moved in place.
  • Building the optimizer and then moving the model. Move the model first, then create the optimizer, as the PyTorch docs advise.
  • Calling .numpy() on a CUDA tensor without .cpu() first.
  • Timing GPU code without torch.cuda.synchronize(), which measures only the launch.
  • Using float16 autocast without a GradScaler, and watching the loss turn into NaN.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does x = x.to(device) do for a tensor x on the CPU and device = cuda?

    Tensor.to returns a copy on the target device. Without assigning it, your variable still points to the CPU tensor.
  2. 2. Why does float16 training usually need a GradScaler but bfloat16 does not?

    float16 cannot represent numbers much below 6e-8, so tiny gradients become zero. bfloat16 trades precision for float32's range.
  3. 3. Which part of a training step belongs inside the autocast context?

    Autocast wraps the forward pass and loss. backward() runs outside and automatically uses the same types as the forward ops.
  4. 4. You get 'Expected all tensors to be on the same device'. What is the usual cause?

    An operation received tensors from two devices, typically a model on the GPU and a batch still on the CPU.

Summary

  • Choose one device (cuda, mps or cpu) and move the model and every batch to it; mixed devices raise errors.
  • Tensor .to() returns a copy, module .to() works in place; use .cpu() before NumPy.
  • pin_memory=True and non_blocking=True overlap data copies with computation.
  • autocast runs safe operations in 16-bit; bfloat16 needs nothing more, float16 needs a GradScaler.
  • Adam training costs about 16 bytes per parameter plus activations; reduce batch size or accumulate gradients when memory runs out.

Next, find out where the time goes in Making Training Faster.