Skip to content
Tutorials Center
My learning

Topics

Lesson 45 of 45 ยท Practice and Review

PyTorch Cheat Sheet

A one-page PyTorch cheat sheet: tensor creation, shapes, autograd, nn layers, losses, optimizers, a training loop template, device handling and saving models.

This page puts the whole track on one screen: the calls you use every day, grouped by task, in compact snippets you can scan in seconds. Keep it open while you write code, while working through the practice exercises, or right before an interview.

PyTorch cannot run on this site, so every PyTorch snippet below is read-only. Two live editors at the end show the NumPy equivalents of the shape operations and a training-loop template that you can run. They are stand-ins for the library, not the library itself. Each section links to the lesson that explains the topic in depth.

The five-step pattern

Data Dataset and DataLoader.
Model An nn.Module.
Loss How wrong are we?
Optimizer Update the parameters.

Then repeat per batch: zero, forward, backward, step.

1 / 2

Tensors: creating them

See Your First Tensors.

Python
import torch

torch.tensor([[1., 2.], [3., 4.]])      # from Python data (copies)
torch.zeros(2, 3)                       # also ones, full((2, 3), 7.0), empty
torch.arange(0, 10, 2)                  # 0, 2, 4, 6, 8
torch.linspace(0, 1, steps=5)           # 5 evenly spaced values
torch.eye(3)                            # identity matrix
torch.rand(2, 3)                        # uniform [0, 1)
torch.randn(2, 3)                       # standard normal
torch.randint(0, 10, (2, 3))            # random integers
torch.zeros_like(x)                     # same shape, dtype and device as x
torch.manual_seed(0)                    # reproducible random numbers
torch.from_numpy(arr)                   # shares memory with the NumPy array
x.numpy()                               # CPU tensor to array, shares memory
x.item()                                # one-element tensor to a Python number

Attributes and dtypes

See Dtypes, Precision and NumPy Interop.

Python
x.shape            # torch.Size([2, 3]), also x.size()
x.dtype            # torch.float32 is the default for floats
x.device           # cpu, cuda:0, mps
x.ndim, x.numel()  # number of dimensions, total elements

x.to(torch.float64)        # change dtype (returns a new tensor)
x.float(); x.long()        # shortcuts: float32, int64
x.to("cuda")               # change device
x.to(device, dtype=torch.float16)
dtype Typical use
float32 default for model weights and inputs
float16, bfloat16 mixed precision on GPUs
int64 (long) class labels, indices, embedding inputs
bool masks

Indexing and shapes

See Indexing, Slicing and Reshaping.

Python
x[0], x[:, 1], x[..., -1]          # row, column, last element of last dim
x[x > 0]                           # boolean mask (returns a copy, 1-D)
x.view(2, -1)                      # no copy, needs compatible layout
x.reshape(2, -1)                   # view if possible, otherwise copy
x.permute(0, 3, 1, 2)              # reorder dims, e.g. NHWC to NCHW
x.transpose(0, 1); x.T             # swap two dims (T is for 2-D)
x.unsqueeze(0); x.squeeze()        # add or drop size-1 dims
x.flatten(start_dim=1)             # keep batch dim, flatten the rest
x.contiguous()                     # standard memory layout
torch.cat([a, b], dim=0)           # join along an existing dim
torch.stack([a, b], dim=0)         # join along a NEW dim
x.expand(4, -1)                    # broadcast without copying

Math and reductions

See Tensor Math and Broadcasting.

Python
a + b, a * b, a / b, a ** 2        # elementwise, with broadcasting
a @ b; torch.matmul(a, b)          # matrix product (batched for 3-D+)
x.sum(), x.mean(dim=0), x.max(dim=1)
x.sum(dim=1, keepdim=True)         # keep the reduced dim for broadcasting
x.argmax(dim=-1)                   # index of the maximum
x.std(), x.var(), x.norm()
torch.clamp(x, min=0, max=1)
torch.where(x > 0, x, 0.0)
torch.einsum("bij,bjk->bik", a, b) # batched matmul
x.add_(1)                          # trailing underscore means in place

Autograd

See Autograd in Practice.

Python
w = torch.randn(3, requires_grad=True)
loss = (w * x).sum()
loss.backward()                    # fills w.grad (accumulates!)
w.grad.zero_()                     # reset a gradient by hand

with torch.no_grad():              # no graph recording (evaluation, updates)
    y = model(x)
with torch.inference_mode():       # stricter and faster than no_grad
    y = model(x)
y = z.detach()                     # cut from the graph, shares data
loss.backward(retain_graph=True)   # keep the graph for a second backward
torch.autograd.grad(loss, w)       # gradients without touching .grad

nn: layers and models

See Modules, Parameters and Layers.

Python
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.body = nn.Sequential(
            nn.Linear(784, 128), nn.ReLU(), nn.Dropout(0.2),
            nn.Linear(128, 10),
        )

    def forward(self, x):
        return self.body(x.flatten(1))

model = Net()
list(model.parameters())               # what the optimizer receives
model.state_dict()                     # parameters plus buffers
sum(p.numel() for p in model.parameters())
model.train(); model.eval()            # switch layer behavior
Layer Purpose
nn.Linear(i, o) fully connected, weight shape (o, i)
nn.Conv2d(c_in, c_out, k, stride, padding) 2-D convolution on (N, C, H, W)
nn.MaxPool2d(k), nn.AdaptiveAvgPool2d(1) downsampling
nn.BatchNorm1d/2d(c), nn.LayerNorm(d) normalization
nn.Dropout(p) regularization, off in eval()
nn.Embedding(n, d) integer ids to vectors
nn.LSTM, nn.GRU recurrent layers (batch_first=True)
nn.TransformerEncoderLayer(d, heads) self-attention block

Output size of a convolution along one dimension: floor((n + 2p - k) / s) + 1. See Convolutions and CNNs.

Activations and losses

See Activation Functions and Loss Functions.

Task Final layer Loss Targets
Regression none nn.MSELoss() float, same shape as output
Multi-class none (logits) nn.CrossEntropyLoss() long class indices
Binary none (logit) nn.BCEWithLogitsLoss() float 0.0 / 1.0
Python
torch.relu(x); nn.GELU(); torch.sigmoid(x); torch.tanh(x)
torch.softmax(logits, dim=-1)       # only for reading probabilities
logits.argmax(dim=-1)               # predicted class, no softmax needed

Optimizers and schedules

See Optimizers and Learning Rate Schedules and Gradient Clipping.

Python
import torch.optim as optim

opt = optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
opt = optim.Adam(model.parameters(), lr=1e-3)
opt = optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)

sched = optim.lr_scheduler.StepLR(opt, step_size=10, gamma=0.1)
sched = optim.lr_scheduler.CosineAnnealingLR(opt, T_max=50)
sched = optim.lr_scheduler.OneCycleLR(opt, max_lr=1e-2, total_steps=1000)

torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

Data

See Datasets and DataLoaders.

Python
from torch.utils.data import Dataset, DataLoader, random_split

class MyData(Dataset):
    def __init__(self, x, y):
        self.x, self.y = x, y
    def __len__(self):
        return len(self.x)
    def __getitem__(self, i):
        return self.x[i], self.y[i]

train_set, val_set = random_split(MyData(X, Y), [0.8, 0.2])
loader = DataLoader(train_set, batch_size=64, shuffle=True,
                    num_workers=4, pin_memory=True)

The training loop template

See The Complete Training Loop.

Python
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

for epoch in range(num_epochs):
    model.train()
    for x, y in train_loader:
        x, y = x.to(device), y.to(device)
        opt.zero_grad()
        loss = loss_fn(model(x), y)
        loss.backward()
        opt.step()
    sched.step()

    model.eval()
    correct = 0
    with torch.no_grad():
        for x, y in val_loader:
            x, y = x.to(device), y.to(device)
            correct += (model(x).argmax(dim=-1) == y).sum().item()
    print(epoch, correct / len(val_set))

Devices and mixed precision

See GPUs, Devices and Mixed Precision.

Python
torch.cuda.is_available(); torch.backends.mps.is_available()
x = x.to(device)                       # tensors return a new object
model.to(device)                       # modules change in place
x.to(device, non_blocking=True)        # with pin_memory=True loaders

scaler = torch.amp.GradScaler()        # float16 only; bfloat16 does not need it
with torch.autocast(device_type="cuda", dtype=torch.float16):
    loss = loss_fn(model(x), y)
scaler.scale(loss).backward()
scaler.step(opt)
scaler.update()

Saving and loading

See Saving and Loading Models.

Python
torch.save(model.state_dict(), "model.pt")             # save weights only
model.load_state_dict(torch.load("model.pt", weights_only=True))

torch.save({"model": model.state_dict(),               # a full checkpoint
            "opt": opt.state_dict(),
            "epoch": epoch}, "ckpt.pt")

Create the model with the same architecture before loading, and call model.eval() before inference.

Try it: NumPy equivalents of the shape operations

The array operations below behave like their tensor twins, which makes them a safe way to rehearse shapes. Names differ slightly: permute is transpose, unsqueeze is np.expand_dims, and keepdim is keepdims.

Python Shape operations in NumPy (stand-in for tensors)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Try it: a training loop template

This loop has the same structure as the PyTorch template above, written with NumPy and a hand-computed gradient. The comments name the PyTorch call that each line replaces.

Python The loop skeleton (NumPy stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which targets does nn.CrossEntropyLoss expect for a 10-class problem?

    CrossEntropyLoss takes raw logits of shape (N, 10) and class indices of dtype long with shape (N,).
  2. 2. In which order should the calls of one training iteration appear?

    Gradients are cleared, then computed by backward(), then consumed by step(). Clearing just before backward() is the usual place.
  3. 3. What is the main difference between torch.stack and torch.cat?

    Stacking three (2, 3) tensors gives (3, 2, 3); concatenating them along dim 0 gives (6, 3).
  4. 4. Which call reduces the risk of running arbitrary code when you load a checkpoint?

    weights_only=True restricts unpickling to tensors and simple containers, which avoids executing arbitrary code stored in a pickle.

Summary

  • Create tensors with tensor, zeros, rand, arange and from_numpy; check shape, dtype and device first when something fails.
  • view shares memory, reshape may copy, permute reorders, stack adds a dimension and cat does not.
  • Autograd accumulates into .grad; wrap evaluation in model.eval() and torch.no_grad().
  • Pick the loss by task and feed it raw logits; labels for classification are long.
  • A training iteration is: move data, zero_grad, forward, loss, backward, step.
  • Save state_dicts, load them with weights_only=True, and keep model and data on the same device.

Back to Practice Exercises with Solutions to put it to use, or revisit the track from What Is PyTorch?.