Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 73 ยท Training in Practice

Automatic Differentiation From Scratch

Build reverse-mode automatic differentiation: a scalar Value class with a computation graph, topological order, backward(), and a tensor version.

In the mini framework every layer needed a hand-written backward method. That is manageable for Linear and ReLU, but every new operation (a convolution, an attention block, a custom loss) would mean deriving and coding another gradient, and one slip silently ruins training. Modern frameworks avoid this with automatic differentiation, usually shortened to autodiff or autograd: the program records the operations it performs, and the gradient of any result is computed mechanically from that record.

In this lesson you will build a working autodiff engine in about fifty lines of Python, train a small network with it, and then see how the same idea scales from single numbers to tensors, which is exactly what PyTorch's autograd does.

How autodiff works

Record Every operation remembers its inputs and how to differentiate itself.
Graph Operations form a directed graph from inputs to the result.
Reverse Walk the graph backward, applying the chain rule at each node.
Accumulate Each node adds the gradients flowing in from all its uses.

No symbolic algebra and no finite differences: just local rules and the chain rule.

1 / 2

Local rules, global gradient

Every program, however large, is built from tiny operations: add, multiply, exp, tanh, and so on. The derivative of each one with respect to its inputs is simple and known. The chain rule says the derivative of the final output with respect to an early value is the product of these local derivatives along each path from that value to the output, summed over all paths.

Take e = a * b + c, with a = 2, b = -3, c = 10. Break it up as d = a * b and e = d + c. Written as a graph:

flowchart LR
    a["a = 2"] --> d["d = a*b = -6"]
    b["b = -3"] --> d
    d --> e["e = d+c = 4"]
    c["c = 10"] --> e

A computation graph: inputs on the left, the result on the right. Each arrow is a dependency.

Going backward from e: the gradient of e with respect to itself is 1. An addition passes the incoming gradient to both inputs unchanged, so de/dd = 1 and de/dc = 1. A multiplication sends each input the incoming gradient times the other input, so de/da = 1 * b = -3 and de/db = 1 * a = 2. Four tiny rules, and the whole gradient is done. This is reverse-mode autodiff, the mode that suits machine learning: one backward sweep gives the gradient of one output (the loss) with respect to every input (all the parameters), no matter how many there are.

A scalar Value class

We need an object that holds a number and a gradient, remembers which Values produced it, and knows how to push its gradient to them. That last part is a small function, _backward, created inside each operation, closing over the inputs and the output.

Two details matter:

  1. Gradients accumulate with +=. If a value is used in two places, both contribute to its gradient. The chain rule sums over paths, so x * x must give 2x, not x.
  2. Order matters. A node may only push its gradient to its parents after all gradient has arrived at it from everything that uses it. Running the nodes in reverse topological order (children before parents) guarantees that. A depth-first walk that appends a node after visiting its parents produces a topological order; reversing it gives the backward order.
Python The Value class, checked against finite differences

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Autodiff and finite differences agree to six digits, but notice the difference in cost: the numeric check re-evaluates the function twice per input, while one backward() call produced all three derivatives. With millions of parameters, only the backward sweep is affordable.

Training a network with it

Because Value supports +, *, powers and tanh, you can write a neuron as ordinary Python and get gradients for free. The network below has 2 inputs, 6 tanh hidden neurons and 1 tanh output, and learns to separate two interleaved half-moons. The loss is the squared error against targets of -1 and +1. Each training step builds a fresh graph with the forward pass, calls loss.backward(), updates every weight by -lr * grad, and zeroes the gradients. Compare this with the hand-derived training loop from Your First Trained Network: nowhere do we write a derivative.

Python A neural network trained with scalar autodiff

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss falls from about 0.94 to 0.33 and the small network classifies 21 of the 24 points correctly after 80 plain gradient steps; more steps would improve it further. It is slow compared with NumPy, since every single multiplication creates a Python object, which is exactly why real frameworks apply the same idea to whole arrays.

From scalars to tensors

A tensor autodiff engine has the same structure; only the local rules change from number formulas to array formulas. The node stores a NumPy array, and each operation's backward rule is the matrix-form gradient you already know from Backpropagation in NumPy. For C = A @ B, the backward rules are dA = dC @ B.T and dB = A.T @ dC.

One extra concern appears: broadcasting. Adding a bias vector of shape (3,) to a matrix of shape (5, 3) silently repeats it across rows, so in the backward pass the gradient must be summed back over the repeated axes to match the bias's shape. A tensor engine therefore needs an unbroadcast helper.

Python A tiny tensor autodiff engine

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The linear model recovers the weights 1.5, -2.0, 0.5 and the bias 0.7 that generated the data, and the two-layer network's gradients, including the bias with its broadcasting, match finite differences.

How PyTorch autograd works

PyTorch is this design with industrial engineering. A tensor created with requires_grad=True takes part in the graph; every operation on such tensors creates a node that stores a grad_fn (the local backward rule); calling loss.backward() sorts the graph topologically and runs the rules in reverse, accumulating into each leaf's .grad. The code below is read-only because PyTorch is not available here, but each line has a counterpart in what you just wrote.

Python
# Read-only: PyTorch is not available in the browser runner
import torch

w = torch.tensor([[1.0], [2.0]], requires_grad=True)   # a leaf that records gradients
x = torch.tensor([[3.0, 4.0]])
loss = ((x @ w - 10.0) ** 2).mean()   # builds the graph as it runs
loss.backward()                       # topological sort, then chain rule in reverse
print(w.grad)                         # d loss / d w, accumulated into .grad
w.grad.zero_()                        # accumulation is why optimizers zero gradients

A few behaviors follow directly from the design:

  • The graph is rebuilt on every forward pass ("define by run"), so ordinary Python control flow such as loops and if statements just works.
  • Gradients accumulate into .grad, hence the need for zero_grad().
  • Inference does not need the graph. Wrapping code in torch.no_grad() skips recording, saving memory.
  • Memory grows with the graph. Every intermediate value needed by a backward rule is kept alive until backward() runs, which is why large models need so much GPU memory during training.

Reverse mode is efficient when there are many inputs and one output (millions of weights, one loss). Forward-mode autodiff has the opposite profile and is used for other problems, such as computing a few derivatives of a function with many outputs.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why do the _backward functions use += to update gradients?

    For y = x * x, x appears twice. Each use contributes x to its gradient, and the total is 2x. Plain assignment would keep only the last contribution.
  2. 2. Why does backward() process nodes in reverse topological order?

    A node can only push its gradient to its parents after every downstream node has sent its share, which reverse topological order guarantees.
  3. 3. In the graph e = a*b + c with a=2, b=-3, c=10, what is de/da?

    The addition passes gradient 1 to the product node, and the product sends each input the incoming gradient times the other input: 1 * b = -3.
  4. 4. Why does a tensor autodiff engine need an unbroadcast step?

    A bias of shape (3,) added to a (5, 3) matrix is used five times, so its gradient is the sum of five contributions with the bias's own shape.

Summary

  • Autodiff records each operation and its local derivative rule, then applies the chain rule in reverse over the resulting graph.
  • Reverse mode gives the gradient of one scalar output with respect to all inputs in roughly the cost of a few forward passes.
  • A Value class needs parents, a gradient, and a _backward closure per operation; gradients accumulate with +=.
  • Backward runs in reverse topological order so each node's gradient is complete before it is passed on.
  • The tensor version uses matrix-form rules and must sum gradients back over broadcast axes.
  • PyTorch's autograd is this design: requires_grad, grad_fn, backward() and accumulated .grad.

Next: Images as Tensors.