Skip to content
Tutorials Center
My learning

Topics

Lesson 50 of 73 ยท Frameworks and Production

PyTorch Essentials

Learn PyTorch by mapping it onto the NumPy network you already built: tensors, autograd, nn.Module, DataLoader, optimizers, the training loop and GPUs.

You now know how a neural network works from the inside: you wrote the forward pass, the backward pass, the optimizer and the training loop yourself. In practice almost nobody does that for real projects. They use a framework, and the most popular one for research and a large share of production work is PyTorch. The good news is that a framework does not add new ideas. Every PyTorch concept is a polished version of something you have already built.

This lesson walks through the PyTorch building blocks one at a time and puts each next to its NumPy equivalent. The PyTorch snippets are read-only, because PyTorch is not available in the in-browser runner. The NumPy versions below them run live, and they are shaped exactly like the PyTorch code, so reading one teaches you the other.

PyTorch in four blocks

Tensor An array that can live on a GPU and track gradients.
Autograd Records operations and runs backpropagation for you.
nn.Module A layer or a whole model that owns its parameters.
DataLoader Shuffles data and hands out mini-batches.

Add an optimizer that updates the parameters, and that is the whole toolkit.

1 / 2

Tensors

A PyTorch tensor is an n-dimensional array, very close to a NumPy ndarray: it has a shape, a data type and the same broadcasting rules. It has two extras. It can be moved to a GPU, and it can record the operations applied to it so that gradients can be computed later.

Python
import torch

x = torch.arange(12, dtype=torch.float32).reshape(3, 4)
print(x.shape, x.dtype)          # torch.Size([3, 4]) torch.float32
print(x.sum(dim=0))              # NumPy calls the argument "axis"
w = torch.ones(4, 2)
print((x @ w).shape)             # torch.Size([3, 2])
print(x[:, 1:3])                 # slicing works as in NumPy

The vocabulary differs a little: dim instead of axis, torch.cat instead of np.concatenate, .view or .reshape instead of .reshape. Conversion is cheap: torch.from_numpy(a) and t.numpy() share memory on the CPU. The default float type is float32, the same size you want for neural networks. If shapes still feel shaky, revisit Layers and the Forward Pass.

Here is the NumPy side of the same ideas, plus the thing autograd automates: the gradient of a one-weight loss, computed by hand and verified numerically.

Python Tensors and one gradient, in NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In PyTorch you would write w = torch.tensor(2.0, requires_grad=True), compute the loss, call loss.backward(), and read w.grad. The value would be -24.0, the same number the analytic line prints.

Autograd

PyTorch's requires_grad=True flag tells a tensor to take part in a computational graph: every operation on it is recorded, and calling .backward() on a scalar result walks that graph in reverse, applying the chain rule at each node. This is the same procedure as Backpropagation by Hand, automated for any combination of operations. The lesson on Automatic Differentiation From Scratch builds a small version.

Python
w = torch.tensor(2.0, requires_grad=True)
loss = (w * 3.0 - 10.0) ** 2
loss.backward()
print(w.grad)                    # tensor(-24.)

Three behaviors catch beginners:

  • Gradients accumulate. Each backward() call adds to .grad. That is why every training step starts by clearing the gradients, with optimizer.zero_grad().
  • Evaluation needs no graph. Wrap inference in with torch.no_grad(): so PyTorch does not record operations, saving memory and time.
  • Only scalars start a backward pass. Losses are averaged over the batch into a single number first.

Modules

torch.nn.Module is the base class for everything with parameters: a single layer, a block, or a full network. A module declares its layers in __init__, describes the computation in forward, and PyTorch finds every parameter automatically.

Python
import torch.nn as nn

class Classifier(nn.Module):
    def __init__(self, n_in, n_hidden, n_out):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(n_in, n_hidden),
            nn.ReLU(),
            nn.Linear(n_hidden, n_out),
        )

    def forward(self, x):
        return self.net(x)          # raw scores (logits), no softmax

model = Classifier(2, 16, 2)
print(sum(p.numel() for p in model.parameters()))   # 82 parameters

nn.Linear(2, 16) holds a weight matrix and a bias, like the dense layer in Backpropagation in NumPy. nn.Sequential chains layers, model.parameters() yields every weight and bias for the optimizer, and model.train() / model.eval() switch layers such as dropout and batch norm between their two behaviors. Loss functions are modules too: nn.CrossEntropyLoss() takes logits and integer labels and applies a stable log-softmax internally, so you must not add a softmax layer yourself (see Loss Functions).

Data loading

A Dataset answers "how many examples, and what is example i"; a DataLoader wraps it and serves shuffled mini-batches, optionally using several worker processes.

Python
from torch.utils.data import TensorDataset, DataLoader

ds = TensorDataset(X_tensor, y_tensor)
loader = DataLoader(ds, batch_size=32, shuffle=True)
for xb, yb in loader:            # one epoch
    ...

This is the batch generator from Mini-Batches, Epochs and Data Loading with a standard interface.

Optimizers

torch.optim.SGD(model.parameters(), lr=0.1) stores the parameter list. optimizer.step() reads every .grad and updates the parameter in place. torch.optim.Adam and AdamW are drop-in replacements with the update rules from Momentum, RMSProp and Adam. Learning-rate schedules live in torch.optim.lr_scheduler.

The training loop

Put the pieces together and you get the loop that nearly every PyTorch project contains:

Python
model = Classifier(2, 16, 2)
loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.SGD(model.parameters(), lr=0.1)

for epoch in range(30):
    model.train()
    for xb, yb in loader:
        opt.zero_grad()              # 1. clear old gradients
        logits = model(xb)           # 2. forward pass
        loss = loss_fn(logits, yb)   # 3. loss on this batch
        loss.backward()              # 4. backpropagation
        opt.step()                   # 5. parameter update
flowchart LR
    A["zero_grad()"] --> B["model(xb)"]
    B --> C["loss_fn(logits, yb)"]
    C --> D["loss.backward()"]
    D --> E["opt.step()"]
    E --> A

The five-step PyTorch training step; it repeats for every mini-batch.

Now the same loop in NumPy, with classes named like their PyTorch counterparts. The Linear layer accumulates into dW and db, so you can see exactly why zero_grad is needed.

Python A PyTorch-shaped training loop in NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss falls and the accuracy climbs to about 0.94 after 40 epochs. The network has 82 parameters, matching the count printed for the PyTorch version above. If you delete the opt.zero_grad() line and rerun, the gradients pile up across batches and training goes wrong; that is the NumPy-sized version of the most common PyTorch mistake.

Devices and the GPU

GPUs run thousands of multiply-adds in parallel, which is why deep learning relies on them. In PyTorch, location is explicit: both the model and its data must be on the same device.

Python
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)

for xb, yb in loader:
    xb, yb = xb.to(device), yb.to(device)   # move each batch
    ...

Common device errors are "expected all tensors to be on the same device" (you forgot to move the batch) and out-of-memory (lower the batch size or model size). Calling .item() or .cpu().numpy() brings a result back to the CPU for printing or plotting. Saving and loading is covered in Deploying and Serving Models.

When you meet unfamiliar PyTorch code, look for the five-step loop first. Everything else is the model definition and the data pipeline.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does a PyTorch training step begin with optimizer.zero_grad()?

    Each backward() call accumulates into .grad. Without clearing, gradients from earlier batches would mix into the current update.
  2. 2. What should the final layer of a classifier feed into nn.CrossEntropyLoss?

    CrossEntropyLoss applies a numerically stable log-softmax itself, so it expects raw scores. An extra softmax would squash them twice.
  3. 3. What does loss.backward() do?

    backward() only computes gradients. The optimizer's step() is what changes the parameters.
  4. 4. You get an error that tensors are on different devices. What is the usual fix?

    The model and its inputs must live on the same device, so each batch is moved to wherever the model is.
  5. 5. When you evaluate a trained model, why wrap the code in torch.no_grad()?

    Without it PyTorch keeps intermediate values for a backward pass that never happens.

Summary

  • A tensor is a NumPy-like array that can move to a GPU and record operations for gradients.
  • Autograd builds a graph during the forward pass and backward() applies the chain rule through it; gradients accumulate until cleared.
  • nn.Module owns parameters and defines forward; nn.Sequential, nn.Linear and nn.ReLU mirror the layer classes you wrote.
  • DataLoader serves shuffled mini-batches; optimizers update parameters from their gradients.
  • Every training step is: zero_grad, forward, loss, backward, step.
  • Model and data must be on the same device; use no_grad and eval() for inference.

Next: TensorFlow and Keras Essentials.