Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 45 ยท Autograd

Autograd in Practice

Use PyTorch autograd with confidence: requires_grad, backward(), gradient accumulation, zero_grad, no_grad, detach and retain_graph, with live stand-ins.

In the previous lesson you saw how a graph of operations and the chain rule produce gradients. PyTorch's autograd engine does that bookkeeping for you, and day to day you only touch a handful of switches: which tensors track gradients, when to call backward(), where the results land, and how to turn tracking off. Most beginner bugs in training loops (gradients that pile up, graphs that are freed too early, memory that never shrinks) come from misunderstanding exactly these switches.

This lesson walks through each one with real PyTorch code. PyTorch itself cannot run on this site, so those snippets are read-only. To make the behavior something you can poke at, the live editors use a tiny scalar autograd class written in plain Python with the same rules: accumulating gradients, leaf versus non-leaf tensors, no_grad, detach and a graph that is freed after one backward pass. That class is a stand-in, not PyTorch, but the rules it enforces are the ones PyTorch enforces.

Four switches of autograd

requires_grad Marks a leaf that should collect gradients.
backward() Walks the graph and fills in .grad.
Accumulation New gradients are added to old ones.
no_grad / detach Stop recording the graph.

Everything else in this lesson is a detail of these four.

1 / 2

requires_grad and the first backward pass

A tensor only takes part in autograd if it has requires_grad=True. You set it on the tensors you want to learn, usually the weights. Anything computed from such a tensor is tracked automatically.

Python
import torch

w = torch.tensor(3.0, requires_grad=True)   # a leaf we want a gradient for
x = torch.tensor(2.0)                       # plain data, no tracking

y = w * w * x          # y = w^2 * x, so dy/dw = 2 * w * x = 12
print(y)               # tensor(18., grad_fn=<MulBackward0>)

y.backward()           # run backpropagation from y
print(w.grad)          # tensor(12.)
print(x.grad)          # None, x never asked for a gradient

Three things happened. The multiplication produced y with a grad_fn attached (the recipe to run backward from that node). y.backward() walked from y to the leaves applying the chain rule. And the result was stored in w.grad, a tensor of the same shape as w.

backward() with no arguments only works on a scalar output, because "the gradient of a vector with respect to a vector" is a whole matrix. For a non-scalar output, either reduce it to a scalar (loss = y.sum()) or pass a gradient argument of the same shape as y that says how much each output element counts. In practice your loss is almost always a scalar, so this rarely comes up.

Leaf and non-leaf tensors

A leaf is a tensor that has no history: you created it yourself (torch.tensor, torch.randn, a Parameter), or it does not require gradients. A non-leaf is the result of an operation on tracked tensors. You can ask any tensor:

Python
a = torch.tensor(1.5, requires_grad=True)
b = a * 4
print(a.is_leaf, a.grad_fn)   # True None
print(b.is_leaf, b.grad_fn)   # False <MulBackward0 object at ...>

b.backward()
print(a.grad)                 # tensor(4.)
print(b.grad)                 # None, with a warning about reading .grad of a non-leaf

By default PyTorch keeps .grad only for leaves, because intermediate gradients are rarely needed and storing them wastes memory. If you do want one (for debugging, say), call b.retain_grad() before backward(). Another rule worth knowing: you may not modify a leaf that requires grad in place (w += 1 raises an error), because autograd may need its old value. Inside an optimizer step this is done under no_grad, which is why optimizers can update weights.

Here is the same idea in the stand-in. The Var class records, for every result, its parents and a small function that turns the output gradient into parent gradients. Run it and compare each line with the PyTorch above.

Python A scalar autograd stand-in with the same rules

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Gradients accumulate

Look at the output of the stand-in: a second backward() did not replace w.grad, it added to it, giving 24 instead of 12. PyTorch behaves the same way, on purpose. Accumulation is useful when you want to simulate a larger batch by summing the gradients of several small ones, but it means you must clear gradients between optimization steps or every step uses a growing sum of all earlier gradients.

The canonical order inside a training loop is therefore:

Python
for xb, yb in loader:
    optimizer.zero_grad()          # 1. clear old gradients
    loss = loss_fn(model(xb), yb)  # 2. forward pass builds a fresh graph
    loss.backward()                # 3. fill .grad on every parameter
    optimizer.step()               # 4. update the parameters using .grad

Each pass through the loop builds a brand-new graph in step 2, so the graph itself never carries over; only .grad does, and zero_grad is what clears it. By default optimizer.zero_grad() sets gradients to None instead of filling them with zeros (the argument is set_to_none=True), which saves a memory write. Calling zero_grad before or after step is fine as long as it happens once per iteration before the next backward(). For single tensors you can write w.grad.zero_(). See optimizers for the full picture.

Freeing the graph and retain_graph

After backward() PyTorch frees the intermediate buffers saved for the backward pass, since normally you never need them again. Running backward() a second time through the same graph therefore fails:

Python
w = torch.tensor(3.0, requires_grad=True)
y = (w * w).sum()
y.backward()
y.backward()
# RuntimeError: Trying to backward through the graph a second time ...

If you really need to go backward twice from the same output (for example, to compute two different losses that share a large subgraph), pass retain_graph=True to the first call. Be careful: a graph kept alive keeps its memory alive too. If you meet this error inside a loop, the real cause is usually a tensor carried over from the previous iteration, such as an RNN hidden state, that still points at an old graph. The fix is to detach() it, not to sprinkle retain_graph=True everywhere.

Turning tracking off: no_grad, inference_mode and detach

Recording a graph costs time and memory. When you do not need gradients (evaluation, inference, updating weights by hand), tell PyTorch not to record.

Python
with torch.no_grad():
    z = w * 2
print(z.requires_grad)   # False

@torch.no_grad()
def predict(model, x):
    return model(x)

with torch.inference_mode():       # a stricter, faster no_grad for pure inference
    out = model(x)

torch.no_grad() is the everyday choice: results computed inside have requires_grad=False and no grad_fn. torch.inference_mode() goes further: tensors created inside are "inference tensors" that skip extra bookkeeping (version counters), making it a little faster. The price is that such tensors cannot later be used in operations that autograd needs to record, so use it for serving and evaluation, not for tensors you might train on afterwards.

detach() is different: it does not change the global mode, it returns a new tensor that shares the same data but has no history. Use it when one value should be treated as a constant, or to move a result out of the graph for logging or NumPy.

Python
h = (w * 5).detach()      # h is a constant now; no gradient flows through it
loss = (w * h).sum()      # gradient w.r.t. w is just h, nothing from inside h

w.numpy()                 # RuntimeError: Can't call numpy() on Tensor that requires grad
w.detach().numpy()        # fine, shares memory with w
loss_value = loss.item()  # a Python float, ideal for logging

Because detach() shares storage, writing into the detached tensor also changes the original values. If you want an independent copy, use w.detach().clone().

The stand-in at the end of the live example above shows both effects: a result computed under no_grad reports requires_grad = False, and the detached copy keeps the number but loses its history.

Checking gradients numerically

How do you know a gradient is right? The oldest trick is to nudge each input a tiny bit and measure the change in the output, the finite-difference (central difference) estimate

df/dw โ‰ˆ (f(w + h) โˆ’ f(w โˆ’ h)) / (2h).

It is slow (two function calls per number), but it needs no calculus at all, so it works as an independent referee for analytic or autograd gradients. The editor below compares the hand-derived gradient of a mean squared error with the numerical estimate. In PyTorch you would compare w.grad against exactly such a number, which is what torch.autograd.gradcheck automates (see Custom Autograd Functions and Hooks).

Python Check an analytic gradient with finite differences

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common mistakes

  • Forgetting zero_grad(): loss still goes down at first, then training behaves strangely because gradients are sums of many steps.
  • Evaluating without no_grad: the code is correct but uses far more memory because every validation batch builds a graph.
  • Calling .item() or .numpy() on tracked tensors in the middle of the graph: .item() is fine for logging; .numpy() needs detach() first, and converting mid-graph breaks gradient flow.
  • Storing the loss tensor in a list (losses.append(loss)) keeps every graph alive. Append loss.item() or loss.detach().
  • In-place edits of tensors that autograd saved for backward raise a "modified by an inplace operation" error. Prefer out-of-place versions (x = x + 1) when in doubt.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. You call loss.backward() twice in a row without any reset, using retain_graph=True the first time. What happens to w.grad?

    Gradients accumulate in .grad. That is why training loops call zero_grad() once per iteration.
  2. 2. Which tensors keep their .grad after backward() by default?

    Non-leaf tensors drop their gradients to save memory. Call retain_grad() on a non-leaf if you want to inspect it.
  3. 3. What does detach() do?

    detach() gives a view of the same data without a grad_fn. It does not copy and does not change the global mode, unlike no_grad().
  4. 4. Why does a loop that keeps hidden states from the previous iteration often fail with 'Trying to backward through the graph a second time'?

    The carried-over tensor links back to the previous iteration's graph. Detach it at the start of each iteration instead of using retain_graph=True.

Summary

  • requires_grad=True marks leaves that should collect gradients; results computed from them get a grad_fn and are tracked.
  • backward() fills .grad on leaves only (unless you call retain_grad()), and needs a scalar or a gradient argument.
  • Gradients accumulate, so call zero_grad() once per training iteration.
  • The graph is freed after backward(); retain_graph=True keeps it, but is rarely the right fix.
  • no_grad() and inference_mode() stop recording for evaluation and inference; detach() cuts a single tensor out of the graph and shares its data.
  • Finite differences are an independent way to verify any gradient.

Next, build the whole engine yourself in Build a Tiny Autograd Tensor.