Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 45 ยท Autograd

Build a Tiny Autograd Tensor

Write a NumPy-backed Tensor with add, mul, matmul, relu and sum, a topological backward pass and broadcasting-aware gradients, then train a small network.

The best way to stop treating autograd as magic is to build a small version of it. In the scalar autograd lesson every node held one number. Real networks work on arrays, so this lesson upgrades the idea: a Tensor class that wraps a NumPy array, remembers which tensors it was computed from, and knows how to pass gradients back to them. About sixty lines are enough for addition, multiplication, matrix multiplication, ReLU and sum, which is all you need to train a small neural network.

Everything here runs live in your browser, because it is plain NumPy. It is a stand-in for PyTorch's engine, not PyTorch: PyTorch is written in C++, supports hundreds of operations, many dtypes and devices, and does far more bookkeeping. But the design, a graph of nodes with a local backward rule each, is the same. The PyTorch code in this lesson is read-only and only shown for comparison.

A Tensor remembers three things

data The NumPy array with the values.
grad An array of the same shape, filled by backward.
parents The tensors it was computed from.

Each result also stores a tiny function that pushes its gradient to its parents.

1 / 2

The shape problem: broadcasting

Scalars only need a + b to give 1 and 1 as local gradients. Arrays add one wrinkle: broadcasting. If you add a bias of shape (3,) to a matrix of shape (4, 3), NumPy silently repeats the bias four times (see Tensor Math and Broadcasting and the NumPy broadcasting lesson). The output gradient has shape (4, 3), but the bias is a single (3,) vector that influenced four rows. Its gradient is therefore the sum of the four rows' gradients.

The general rule: whenever an operand was stretched during the forward pass, sum the incoming gradient over the stretched axes on the way back. That is a few lines:

Python
def unbroadcast(grad, shape):
    while grad.ndim > len(shape):          # axes that were added on the left
        grad = grad.sum(axis=0)
    for axis, size in enumerate(shape):    # axes of size 1 that were stretched
        if size == 1 and grad.shape[axis] != 1:
            grad = grad.sum(axis=axis, keepdims=True)
    return grad

Forgetting this is the number one bug in homemade autograd engines: the code runs and the shapes even look plausible for square matrices, but the gradients are wrong.

The rules for each operation

Every operation creates a new tensor and defines a _backward function that adds to the parents' grad arrays (with +=, because a tensor may be used more than once, as in x * x).

Operation Forward Gradient sent to the parents
a + b a + b g to each, after unbroadcasting
a * b a * b g * b to a, g * a to b, after unbroadcasting
a @ b matrix product g @ b.T to a, a.T @ g to b
relu(a) max(a, 0) g where a > 0, else 0
a.sum() sum of all entries g copied to every entry

The matrix-multiplication rule is worth deriving once. For C = A @ B, each entry C[i, j] is the dot product of row i of A and column j of B. A value A[i, k] affects the whole row C[i, :], with weights B[k, :], so its gradient is the dot product of g[i, :] and B[k, :]. Stacked up, that is g @ B.T. The same argument for B gives A.T @ g. The shapes are a handy check: (n, m) @ (m, p) gives (n, p), and g @ B.T is (n, p) @ (p, m), which is (n, m), the shape of A.

The backward pass

backward() first builds a topological order: a list in which every tensor comes after all of its parents. A depth-first search does that, adding a node only after visiting its parents. Then it sets the output's gradient to ones (the derivative of the output with respect to itself) and walks the list in reverse, calling each node's _backward. When a node is processed, every consumer of that node has already added its contribution, so its grad is complete. That is the entire algorithm.

Here is the whole class with a gradient check. We test against central finite differences (see Autograd in Practice) for a small network sum(relu(x @ W + b)), where b is broadcast over the rows.

Python A NumPy Tensor with backward()

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If the two matches lines print True, the engine is correct for those operations. Notice that b.grad has the bias's own shape (2,) even though the output has shape (4, 2). That is unbroadcast at work. The last lines also show why gradients are added with +=: in x * x + x the same tensor feeds three inputs, and their contributions must be summed to give 2x + 1.

Training a network with it

Gradients are only useful if you do something with them. The next editor trains a one-hidden-layer network, 1 -> 16 -> 1 with ReLU, to fit a sine curve. The loop has exactly the structure you will use in PyTorch: forward pass, loss, backward, update, clear the gradients. We write the update by hand with plain NumPy, which is what an optimizer like SGD does for you.

Python Train a network with your own autograd

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss falls from about 2.27 at step 0 to about 0.006 at the end, and the plotted network follows the sine curve closely. Look at what we did not have to write: no derivative of the network appears anywhere. We only defined the forward pass, and loss.backward() found the gradient of every weight. Changing the architecture, for example adding a second hidden layer, needs no new calculus.

Where PyTorch differs

The same computation in PyTorch is shorter, and the engine is far more capable. This code is read-only; compare it to the stand-in line by line.

Python
import torch

xs = torch.linspace(-3, 3, 60).unsqueeze(1)
ys = torch.sin(xs)

W1 = (torch.randn(1, 16) * 0.8).requires_grad_()
b1 = torch.zeros(16, requires_grad=True)
W2 = (torch.randn(16, 1) * 0.3).requires_grad_()
b2 = torch.zeros(1, requires_grad=True)

for step in range(1500):
    pred = (xs @ W1 + b1).relu() @ W2 + b2
    loss = ((pred - ys) ** 2).mean()
    loss.backward()
    with torch.no_grad():
        for p in (W1, b1, W2, b2):
            p -= 0.02 * p.grad
            p.grad = None

The differences that matter:

  • Operation coverage: our engine has five operations; PyTorch has hundreds, each with a hand-written (or generated) backward rule.
  • Graph lifetime: PyTorch builds a fresh graph per forward pass and frees its buffers after backward(). Our toy keeps the graph alive as long as a reference exists.
  • Leaf gradients accumulate: we zero the gradients by hand each step, as PyTorch requires (p.grad = None).
  • Speed and devices: the same graph can run on a GPU, in half precision, or be compiled.
  • Safety checks: PyTorch tracks tensor versions to detect in-place edits that would silently corrupt a gradient.

Next you will see how to teach PyTorch a brand-new operation by writing its backward rule yourself.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A bias of shape (3,) is added to a matrix of shape (4, 3). What shape does the bias gradient have, and how is it computed?

    The bias was repeated across four rows in the forward pass, so every row contributed to its gradient. The backward pass sums over the broadcast axis, which gives back the bias's own shape (3,).
  2. 2. Why are gradients accumulated with += instead of assigned with = in each _backward function?

    In x * x + x the same tensor is used three times. The total gradient is the sum of the three contributions, 2x + 1.
  3. 3. For C = A @ B, which expression gives the gradient sent back to A, with g being the gradient of C?

    Shapes confirm it: g is (n, p), B.T is (p, m), so g @ B.T is (n, m), the shape of A. The gradient for B is A.T @ g.
  4. 4. Why does backward() visit nodes in reverse topological order?

    In the reversed order every consumer of a node runs before the node itself, so by the time a node's rule runs, all contributions to its grad have been added.

Summary

  • An autograd tensor stores its data, a gradient array of the same shape, its parent tensors and a function that pushes gradient to those parents.
  • Gradients of operations that broadcast must be summed back to the shape of each operand (unbroadcast).
  • a @ b sends g @ b.T to a and a.T @ g to b; ReLU passes gradient where the input was positive.
  • backward() sorts the graph topologically, seeds the output gradient with ones and calls each node's rule in reverse order; gradients are added with +=.
  • A finite-difference check is the quickest way to find a wrong rule.
  • PyTorch follows the same design with many more operations, a freed graph per step, version checks and device support.

Continue with Custom Autograd Functions and Hooks.