Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 45 ยท Autograd

Computational Graphs and the Chain Rule

Understand how autograd works: forward and backward passes, computational graphs and the chain rule, walked through by hand and rebuilt in a tiny Python Value class.

Training a neural network means nudging millions of numbers (the parameters) so a loss gets smaller. To nudge them in the right direction you need the gradient: how much the loss changes when each parameter changes a little. PyTorch computes all of those gradients automatically. This lesson explains how, so that loss.backward() stops being magic.

The answer is two ideas. First, every computation is a graph of small operations. Second, the chain rule from calculus lets you turn the derivatives of those small operations into the derivative of the whole thing. You will walk a derivative by hand and then build a working autograd engine in about 40 lines of Python.

Forward pass: building the graph

Take a small function: L = (a * b + c) ** 2. Break it into steps and give each intermediate value a name:

Text
d = a * b
e = d + c
L = e ** 2

With a = 2, b = 3, c = 1, the forward pass gives d = 6, e = 7, L = 49. Each value remembers which operation made it and from what inputs. That record is the computational graph. PyTorch builds it as your code runs (this is the "eager" part from What Is PyTorch?).

flowchart LR
    a["a = 2"] --> d["d = a * b = 6"]
    b["b = 3"] --> d
    d --> e["e = d + c = 7"]
    c["c = 1"] --> e
    e --> L["L = e ** 2 = 49"]

Each node is a value, each arrow an operation's input. The forward pass flows left to right.

The chain rule, one local step at a time

We want dL/da, dL/db and dL/dc. The chain rule says: to find how L depends on an early value, multiply the local derivatives along the path from that value to L.

Work backwards from L, writing each result as "how much does L change if this value changes by 1":

  1. dL/dL = 1.
  2. L = e ** 2, so the local derivative is 2e = 14. Therefore dL/de = 1 * 14 = 14.
  3. e = d + c. Addition passes the gradient through: dL/dd = 14 and dL/dc = 14.
  4. d = a * b. Multiplication swaps the inputs: dL/da = dL/dd * b = 14 * 3 = 42 and dL/db = dL/dd * a = 14 * 2 = 28.

So the gradients are (42, 28, 14). This reverse sweep is the backward pass, and the method is called reverse-mode automatic differentiation (backpropagation). It is efficient because one backward sweep gives the gradient of the single loss with respect to every input, no matter how many inputs there are.

Two passes

Forward Compute values, record the graph.
Backward Walk the graph, apply the chain rule.

One backward pass gives every gradient.

1 / 2

Why gradients matter

The gradient points in the direction where the loss grows fastest, so stepping against it reduces the loss. With a learning rate lr, one update is param = param - lr * grad. Repeat that thousands of times and the model learns. The next chapters build optimizers and training loops on top of this single step.

Here is the same graph in real PyTorch, read-only:

Python
import torch

a = torch.tensor(2.0, requires_grad=True)
b = torch.tensor(3.0, requires_grad=True)
c = torch.tensor(1.0, requires_grad=True)

L = (a * b + c) ** 2
L.backward()

print(a.grad, b.grad, c.grad)   # tensor(42.) tensor(28.) tensor(14.)

requires_grad=True tells PyTorch to record operations on that tensor. backward() runs exactly the reverse sweep you did by hand.

Check it numerically

You can verify any gradient by nudging an input and watching the output, the finite difference method: (f(x + h) - f(x)) / h. It is too slow for training but perfect for checking.

Python The chain rule by hand, checked numerically

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The analytic and numeric gradients agree (42, 28, 14). The hand calculation is exactly what autograd automates. This is plain Python, not PyTorch.

Build the engine: a scalar Value class

Now we automate the recipe. A Value stores a number, its gradient, the values it came from, and a small function that knows how to push gradient back to those parents. backward() visits the graph in reverse order (a topological sort) and calls each function. This is the same design PyTorch uses, only for single numbers instead of tensors, and written in pure Python for clarity. It is a stand-in, not PyTorch itself.

Python A tiny autograd engine

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The engine prints L = 49.0 and gradients 42.0 28.0 14.0, matching the hand calculation. Look at two details. Each _backward uses +=, because a value used in several places receives gradient from each of them. And backward() starts at the loss with gradient 1 and goes in reverse topological order so every node's gradient is complete before it is passed on.

One value used twice

The += matters whenever a value appears more than once. In L = a * a, the derivative is 2a, and the engine gets it by adding the two contributions.

Python Reused values accumulate gradient

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Because a is both inputs of the multiplication, each side adds 5.0, giving 10.0, the same as 2a. The next lesson shows the same accumulation in real PyTorch, and why you must clear gradients between steps: Autograd in Practice.

Note: Real PyTorch works on whole tensors, not single numbers, and its local rules involve matrices. The structure is identical: record a graph forward, then walk it in reverse applying local derivatives.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the forward pass record that makes backward possible?

    Each result remembers the operation and inputs that created it, forming the graph that backward walks in reverse.
  2. 2. For L = (a * b + c) ** 2 with a=2, b=3, c=1, what is dL/da?

    dL/de = 2e = 14, addition passes it to d, and multiplying by b = 3 gives 14 * 3 = 42.
  3. 3. Why does backward use += when storing gradients?

    When a value feeds multiple operations, the gradients flowing back from each path must be summed.
  4. 4. Why is reverse-mode differentiation efficient for neural networks?

    A network has one scalar loss and millions of inputs; reverse mode yields all their gradients in about the cost of one extra pass.

Summary

  • A computation is a graph of small operations; the forward pass computes values and records the graph.
  • The backward pass applies the chain rule from the loss to the inputs, one local derivative at a time.
  • Addition passes gradient through, multiplication swaps the inputs, and power uses n * x ** (n - 1).
  • Gradients drive learning: step parameters against the gradient to reduce the loss.
  • A scalar Value class with a topological backward pass reproduces what PyTorch does for tensors.
  • Finite differences are a slow but reliable way to check gradients.

Next: use the real thing in Autograd in Practice.