Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 73 ยท Training in Practice

Building a Mini Deep-Learning Framework

Turn loose training code into a small framework: Module, Linear, activations, loss, SGD and Adam classes plus a fit loop, mapped to PyTorch's design.

So far you have written forward and backward passes as loose functions and wired them together by hand. That works for one network, but every new architecture means rewriting the plumbing. Real frameworks solve this with a small set of reusable pieces: layers that know how to run forward and backward, a container that chains them, a loss, an optimizer and a training loop.

In this lesson you will build exactly those pieces in about a hundred lines of NumPy. The design deliberately mirrors PyTorch's nn.Module, nn.Sequential, loss classes and torch.optim, so when you reach PyTorch Essentials the API will feel familiar. One thing PyTorch does that we do not: automatic differentiation. Here each layer carries its own hand-written backward; the next lesson, Automatic Differentiation From Scratch, removes that burden.

The model side

Module Anything with forward, backward and parameters.
Sequential Chains modules, runs backward in reverse.
Loss Returns a number and the first gradient.
1 / 2

Parameters and modules

A parameter is an array that learns, plus a slot for its gradient. A module is any object that turns inputs into outputs (forward), turns the gradient of its output into the gradient of its input (backward), and exposes its parameters so an optimizer can find them.

The contract for backward is the one from Backpropagation in NumPy: it receives dout, the gradient of the loss with respect to the module's output, stores the gradients of its own parameters, and returns the gradient with respect to its input. For that it needs to remember whatever the forward pass computed, so each module caches its input (or output) during forward.

Python
class Module:
    def parameters(self): return []
    def forward(self, x): raise NotImplementedError
    def backward(self, dout): raise NotImplementedError
    def __call__(self, x): return self.forward(x)

Three concrete layers cover most of a multilayer network:

  • Linear: y = x @ W + b. Backward: dW = x.T @ dout, db = dout.sum(0), dx = dout @ W.T.
  • ReLU: y = max(x, 0). Backward passes the gradient only where the input was positive.
  • Tanh: backward multiplies by 1 - y**2.

Sequential holds a list of modules. Forward runs them in order; backward runs them in reverse order, handing each one's dx to the previous one. That reversal is the entire "back" in backpropagation.

The loss is a module too, with one difference: it receives the targets, returns a scalar, and its backward takes no argument because it is where the gradient chain starts. The optimizer receives the list of parameters and implements step() (and zero_grad()). Finally, fit is a plain loop over shuffled mini-batches (see Mini-Batches, Epochs and Data Loading).

The framework in one block

The next example defines everything, adds an Adam optimizer next to SGD (see Momentum, RMSProp and Adam), and then uses a finite-difference check to verify that a whole three-layer model has correct gradients (Gradient Checking). Notice how few lines each class needs.

Python A complete mini framework, with a gradient check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The tiny relative error confirms that the chain of backward calls produces the true gradient. Building bigger models is now only a matter of listing layers in Sequential.

One subtle point: gradients are accumulated with += in the layers, so the loop must reset them with zero_grad() before each backward pass. Forgetting this is a classic bug; PyTorch has the identical gotcha with optimizer.zero_grad().

The optimizer and the fit loop

Optimizers hold state. Adam keeps a running mean m of each gradient and a running mean v of its square, corrects both for the start-up bias, and moves each parameter by lr * m_hat / (sqrt(v_hat) + eps). In the framework that is one class with two lists of arrays. The fit function ties everything together: shuffle indices each epoch, slice a mini-batch, run forward, loss, zero_grad, backward, step, and record the average loss and, optionally, validation accuracy.

Python Training with the framework: spirals and digits

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both optimizers fit the spirals perfectly on the training data (loss near zero, accuracy 1.0; this is a training-set fit, not a generalization test), and the digit classifier passes 84% test accuracy after one epoch and reaches about 98% after thirty. Notice that fit knows nothing about the architecture: it only calls model(x), model.backward(...) and the optimizer. Swapping in a different model needs no changes.

How it maps onto PyTorch

Nothing here is a toy idea. The structure you wrote is the structure of PyTorch, minus automatic differentiation and GPU support. PyTorch cannot run on this site, so the code below is read-only; it is the same training as the digits example above.

Python
# Read-only: PyTorch is not available in the browser runner
import torch
from torch import nn

model = nn.Sequential(nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, 10))
loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=0.005)

for epoch in range(30):
    for xb, yb in loader:                   # shuffled mini-batches
        loss = loss_fn(model(xb), yb)
        opt.zero_grad()
        loss.backward()                     # replaces model.backward(loss_fn.backward())
        opt.step()
Our mini framework PyTorch
Parameter(data) with .grad nn.Parameter / a tensor with requires_grad=True
Module, __call__ nn.Module, forward
Linear, ReLU, Tanh nn.Linear, nn.ReLU, nn.Tanh
Sequential(...) nn.Sequential(...)
SoftmaxCrossEntropy nn.CrossEntropyLoss
SGD, Adam torch.optim.SGD, torch.optim.Adam
opt.zero_grad(), opt.step() identical names
model.backward(loss_fn.backward()) loss.backward()
fit your own loop, or a library such as Lightning

The one real difference is the last-but-one row. In PyTorch every operation records itself as it runs, so loss.backward() can walk that record and apply the chain rule for you; nobody writes per-layer backward methods. That recording is exactly what you will build next.

Design choices worth noticing

  • Caching in forward. Linear saves x; ReLU saves a mask. Backward is impossible without the forward values, which is why training uses more memory than inference.
  • Accumulating gradients. Using += lets one parameter be used twice (weight sharing) and gradients from several backward passes be summed, which is how gradient accumulation across micro-batches works in large-model training.
  • Separation of concerns. The model knows nothing about the optimizer, and the optimizer knows nothing about the model: it sees only a list of parameters. New optimizers, layers and losses plug in without touching existing code.
  • Train and eval modes. Dropout and batch norm would need a training flag on Module, passed down by Sequential, just like model.train() and model.eval() in PyTorch.

When you build your own layer, test it in isolation with a gradient check before putting it into a model. A wrong backward rarely crashes; it just trains badly.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In Sequential, in which order does the backward pass visit the layers?

    The chain rule starts at the loss and moves toward the input, so the last layer's backward runs first and its dx becomes the previous layer's dout.
  2. 2. Why does Linear.forward store its input x?

    The gradient with respect to W is the input transposed times the output gradient, so backward needs the values seen during forward.
  3. 3. What goes wrong if you forget to call zero_grad() before each backward pass in this framework?

    Layers add into .grad with +=. Without resetting, each step uses the sum of all previous gradients.
  4. 4. Which PyTorch call corresponds to model.backward(loss_fn.backward()) in the mini framework?

    loss.backward() runs the chain rule over the recorded computation, replacing hand-written per-layer backward methods.

Summary

  • A small framework needs five pieces: parameters/modules, a container, a loss, an optimizer and a fit loop.
  • Each layer caches what it needs in forward and implements backward(dout) returning the input gradient and storing its parameter gradients.
  • Sequential runs forward in order and backward in reverse; the optimizer only sees a list of parameters.
  • Gradients accumulate, so zero_grad() must run before every backward pass.
  • The design matches PyTorch's nn.Module, nn.Sequential, loss classes and torch.optim; the missing piece is automatic differentiation.

Next: Automatic Differentiation From Scratch.