Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 45 ยท Projects

Project: A Neural Network Without PyTorch

Build a classifier from scratch: a NumPy autograd Tensor, a Module and Linear layers, softmax cross-entropy and Adam, then compare with the PyTorch code.

In this project you rebuild the core of PyTorch's training stack in about 150 lines of NumPy, and then use it to train a real classifier. You write four pieces: an autograd Tensor, a Module system with Linear, ReLU and Sequential layers, a numerically stable softmax cross-entropy loss, and the Adam optimizer. At the end, the same program in PyTorch fits on a page, and you will see that every line of it corresponds to something you built yourself.

PyTorch cannot run on this site, so that final PyTorch program is read-only. Everything you build runs live, because it is plain NumPy. The stand-in is honest about what it is: a teaching engine with five operations, full-batch training and no GPU. It reuses the ideas from Build a Tiny Autograd Tensor, Modules, Parameters and Layers and Optimizers, so skim those first if any part feels new.

Four pieces to build

Tensor Values, gradients and a backward rule.
Modules Layers that own parameters.
Loss Softmax cross-entropy, stable.
Adam Momentum plus per-weight step sizes.

Then train on three tangled spirals.

1 / 2

Piece 1: the Tensor and a fused loss

The Tensor class is the one from the earlier lesson, with mean() added so the loss can average over the batch. Each operation stores a small _backward function, and backward() walks the graph in reverse topological order. Gradients of broadcast operands are summed back to their shape (unbroadcast).

The new piece is the loss. Cross-entropy on logits is -log(softmax(z)[correct class]). You could build it from exp, sum and log operations, but that is wasteful and numerically fragile. Instead we implement it as one operation with a hand-derived backward rule. First subtract the row maximum from the logits (softmax does not change, but exp can no longer overflow). Then, the gradient of the average loss with respect to the logits turns out to be just (probabilities - one_hot) / n: the model is pushed down on the classes it over-predicted and up on the correct class, proportionally to the error. PyTorch's F.cross_entropy fuses the same operations for the same reasons (see Activation and Loss Functions).

Before trusting any engine, check it. The first editor builds a small network 4 -> 5 -> 3 and compares every gradient with central finite differences (see Autograd in Practice).

Python The engine and a gradient check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All four gradients match. The loss of 4.79 at the start is much worse than the ln(3) = 1.10 of a uniform guess, because the random weights were drawn with a large scale and give confident wrong answers; training will shrink that. The last line shows the fused gradient: for logits [2, 0, -1] and a correct class of 0, the gradient is -0.156 for the right class (push it up) and positive for the other two (push them down). The entries sum to zero, as they must for a softmax.

Piece 2: Modules and layers

A neural network is a tree of objects that each own some parameters. PyTorch's nn.Module finds those parameters automatically; ours does the same with a few lines that scan the object's attributes for Tensors, nested Modules and lists of modules. Linear holds a weight matrix and a bias, ReLU has no parameters, and Sequential calls its layers in order.

For the weights we use He initialization: normal values with variance 2 / n_in, which keeps the size of the activations stable through ReLU layers (see Initialization, Normalization and Dropout). PyTorch's nn.Linear uses a different default (a uniform distribution scaled by 1 / sqrt(n_in)), so the numbers will not match exactly, but both are sensible.

Piece 3: Adam

Plain gradient descent uses one step size for every weight. Adam keeps two running averages per weight: m, the average of the gradient (momentum, which smooths noisy directions), and v, the average of the squared gradient (which tells how large the gradient usually is). The step is lr * m_hat / (sqrt(v_hat) + eps): weights with consistently small gradients take relatively larger steps, and weights with huge gradients are tamed. The m_hat and v_hat terms are bias corrections, because both averages start at zero and would otherwise be too small in the first steps. The defaults betas=(0.9, 0.999) and eps=1e-8 are the same as PyTorch's.

Piece 4: the project

The data are three interleaved spiral arms in the plane, with strong angular noise so that the classes overlap a little at the edges. A straight line cannot separate them, and no model can be perfect because of the noise, which makes the numbers realistic. We hold out 120 of the 360 points as a test set. The model is 2 -> 32 -> 32 -> 3 with ReLU activations (1,251 parameters) trained with full-batch Adam.

Python Train the from-scratch network on spirals

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The training loss falls from about 1.20 (a bit above the ln 3 = 1.10 of random guessing) to about 0.15, training accuracy climbs above 92% and test accuracy reaches about 88%. The gap between train and test is the normal sign of a model that has started to memorize some noise (see Overfitting and Regularization). In the right-hand plot, the colored regions show what the network predicts everywhere in the plane: curved, spiral-shaped regions that no linear model could draw. The squares are test points, and the few that fall in the wrong color are the noisy ones near the borders.

Try changing it and compare the plots: a smaller hidden size, a much larger learning rate, or more epochs each tell a different story. Comment out the optimizer.zero_grad() call to see what accumulating gradients do to the loss curve, which is a good way to feel why the call exists.

The same program in PyTorch

Here is the PyTorch version. It is read-only. Match each part to what you wrote:

Python
import torch
from torch import nn

torch.manual_seed(0)
X_train = torch.tensor(X_train, dtype=torch.float32)       # shape (240, 2)
y_train = torch.tensor(y_train, dtype=torch.long)          # class indices
X_test = torch.tensor(X_test, dtype=torch.float32)
y_test = torch.tensor(y_test, dtype=torch.long)

model = nn.Sequential(                                     # your Sequential
    nn.Linear(2, 32), nn.ReLU(),                           # your Linear and ReLU
    nn.Linear(32, 32), nn.ReLU(),
    nn.Linear(32, 3),
)
loss_fn = nn.CrossEntropyLoss()                            # your fused cross_entropy
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)  # your Adam

for epoch in range(401):
    optimizer.zero_grad()                                  # clear old gradients
    loss = loss_fn(model(X_train), y_train)                # forward and loss
    loss.backward()                                        # your Tensor.backward()
    optimizer.step()                                       # your Adam.step()

model.eval()
with torch.no_grad():
    test_acc = (model(X_test).argmax(dim=1) == y_test).float().mean().item()

Everything you wrote by hand maps to one line. What PyTorch adds is everything around those lines:

  • Many more operations, each with a tested backward rule, on CPU, GPU and other devices.
  • Mini-batches through DataLoader instead of one full-batch step per epoch, which scales to data that does not fit in memory.
  • A graph that is freed after each backward pass, no_grad / inference_mode, and checks against in-place edits that would corrupt gradients.
  • Saving and loading with state_dict (Saving and Loading Models), mixed precision, torch.compile and distributed training.

Where to go from here

  • Add a Tanh module and a Dropout module (in eval mode it must do nothing, in train mode it zeroes random units and rescales). Use the gradient check from the first editor to verify them.
  • Replace full-batch training with mini-batches of 32 shuffled each epoch, and plot the loss again.
  • Add weight decay to Adam (subtract lr * wd * w from each weight) and see how it changes the test accuracy.
  • Implement Tensor.__sub__ and Tensor.exp() so you can express a sigmoid directly, and check its gradient numerically.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is softmax cross-entropy implemented as a single fused operation with the row maximum subtracted?

    Subtracting the maximum keeps exp from overflowing without changing the softmax, and fusing the operations gives a short, exact gradient instead of a long chain of small ones.
  2. 2. What do the two running averages in Adam track?

    m is a moving average of the gradient (momentum) and v is a moving average of its square, used to scale each weight's step.
  3. 3. In the training step, what happens if optimizer.zero_grad() is left out?

    Backward adds into each parameter's grad with +=. Without clearing, every step's gradient includes all previous ones.
  4. 4. The test accuracy (about 88%) is below the training accuracy (about 92%). What does that most likely mean?

    A small gap is normal. A large and growing gap would be the signal to regularize, get more data or stop earlier.

Summary

  • A training framework is four ideas: a tensor that records its history, modules that own parameters, a loss with a stable gradient, and an optimizer that updates weights from gradients.
  • Verify every new operation against finite differences before using it.
  • Softmax cross-entropy has the gradient (probabilities - one_hot) / n with respect to the logits.
  • Adam combines momentum with per-weight step sizes and bias correction; PyTorch's defaults are the same.
  • The PyTorch program is the same loop: zero_grad, forward, loss, backward, step.
  • A small train/test gap is expected; a large one calls for regularization (see the next projects for more practice).

Continue with Project: Handwritten Digit Classifier.