Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 73 ยท Neural Networks From Scratch

Backpropagation in NumPy

Turn backpropagation into matrix code: gradients for dense layers, ReLU and softmax cross-entropy, a reusable Layer class and the shape rules that catch bugs.

In the previous lesson you backpropagated through one example with single numbers. Real networks process a whole batch at once, with weight matrices, so the same rules must be written in matrix form. The good news: they are the same rules, and the shapes of the arrays almost dictate the formulas.

By the end of this lesson you will have three small classes, Dense, ReLU and SoftmaxCrossEntropy, each with a forward and a backward method. They snap together like building blocks and are the core of the framework you will assemble later in Building a Mini Deep-Learning Framework.

Every layer has two methods

forward(x) Compute the output and cache what backward needs.
backward(grad_out) Receive dL/d(output). Return dL/d(input).
Parameter grads Store dL/dW and dL/db on the way.

The returned gradient becomes the previous layer's grad_out.

1 / 2

The dense layer

A dense (fully connected) layer computes Y = X @ W + b, where X has shape (N, d_in), W has shape (d_in, d_out) and b has shape (d_out,). The result Y is (N, d_out). Suppose the layer above has already told us G = dL/dY, with the same shape as Y. We need three gradients.

Gradient for the weights. Each weight W[i, j] multiplies input feature i to produce output j, for every example in the batch. Its gradient is the sum over examples of X[n, i] * G[n, j], which is exactly a matrix product:

TEXT
dL/dW = X.T @ G          shape (d_in, N) @ (N, d_out) = (d_in, d_out)

Gradient for the bias. The bias is added to every row, so it receives the sum of the gradients over the batch:

TEXT
dL/db = G.sum(axis=0)    shape (d_out,)

Gradient for the input, which is passed to the previous layer:

TEXT
dL/dX = G @ W.T          shape (N, d_out) @ (d_out, d_in) = (N, d_in)

You do not need to memorize these. Given the shapes, there is usually only one way to multiply the arrays so the dimensions fit. The test below makes that point: we write the three lines, then verify every shape and compare one entry with a brute-force nudge.

Python A dense layer with forward and backward

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The trick sum(Y * G) is worth remembering: it is a scalar whose gradient with respect to Y is exactly G, so it lets us test a single layer without a full loss.

The golden shape rule: a gradient always has the same shape as the array it is the gradient of. dW matches W, db matches b, dX matches X. If a shape does not match, a transpose or a sum over the batch axis is missing.

ReLU

ReLU(x) = max(0, x) has no parameters and works element by element. Its derivative is 1 where the input was positive and 0 otherwise, so the backward pass is an element-wise multiplication by a mask:

TEXT
dL/dX = G * (X > 0)

Note the *: this is element-wise, not a matrix product. The mask comes from the forward pass, so we cache it.

Softmax with cross-entropy

The last layer turns logits into probabilities and compares them to the labels. Differentiating softmax and cross-entropy separately is messy; combined, the result is beautifully simple. For one example with probabilities p and one-hot label y:

TEXT
dL/dlogits = p - y

For a batch, with the loss averaged over N examples, divide by N:

TEXT
dL/dlogits = (P - Y_onehot) / N

The prediction minus the truth. This is the property that makes cross-entropy so well-behaved, as discussed in Loss Functions. Let us implement both classes and test the softmax-cross-entropy gradient numerically.

Python ReLU and softmax cross-entropy layers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The gradient rows sum to zero because the probabilities sum to 1 and exactly one label entry is 1. That is a handy sanity check.

Chaining layers

With the pieces in place, a network is a list of layers. The forward pass loops through them in order. The backward pass loops through them in reverse, handing each layer the gradient the next layer returned.

flowchart LR
    X["X"] --> D1["Dense 1"] --> R["ReLU"] --> D2["Dense 2"] --> L["Softmax<br/>cross-entropy"]
    L -. "p - y" .-> D2
    D2 -. "dX" .-> R
    R -. "dX" .-> D1

Solid arrows carry activations forward; dashed arrows carry gradients backward, each layer turning dL/d(output) into dL/d(input).

Here is a complete two-layer classifier trained for a few hundred steps on three clusters of points. The update is plain gradient descent from the earlier lesson. The code stays short because each layer owns its own math.

Python Layers chained into a trainable network

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss starts near 1 (a pure guess among three classes would give log 3, about 1.10) and falls to almost zero as the decision regions form.

Shape rules for debugging

When a hand-written backward pass misbehaves, shapes are the first thing to inspect:

Quantity Shape How it is built
Input X (N, d_in) batch on rows
dW same as W X.T @ G, summing over the batch
db same as b G.sum(axis=0)
dX same as X G @ W.T
ReLU gradient same as input G * mask, element-wise

Two classic mistakes: forgetting that the loss is averaged (so the gradient needs a 1/N), and computing db without summing over the batch, which silently broadcasts into a wrong shape. For a systematic safety net, read the next lesson, Gradient Checking.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A Dense layer has X of shape (32, 10) and W of shape (10, 4). What is the shape of dL/dW?

    A gradient has the shape of its parameter. It is computed as X.T @ G, which is (10, 32) @ (32, 4) = (10, 4).
  2. 2. How is the bias gradient computed for a batch?

    The same bias vector is added to each row, so every example contributes to its gradient, and the contributions are summed over the batch axis.
  3. 3. What is the gradient of the averaged softmax cross-entropy loss with respect to the logits?

    Softmax and cross-entropy combine into the simple form p minus y, divided by the batch size because the loss is a mean.
  4. 4. In which order does the backward pass visit the layers?

    The chain rule starts at the loss and moves toward the input, so the last layer goes first.
  5. 5. Why does ReLU's backward method use an element-wise product instead of a matrix product?

    Each output depends only on the matching input, so the Jacobian is diagonal, and multiplying by it reduces to an element-wise multiplication.

Summary

  • A layer implements forward (caching what it needs) and backward (turning the gradient w.r.t. its output into the gradient w.r.t. its input).
  • Dense layer: dW = X.T @ G, db = G.sum(axis=0), dX = G @ W.T.
  • ReLU multiplies the incoming gradient by the cached mask X > 0.
  • Softmax with cross-entropy gives (P - onehot) / N at the logits.
  • A network is a list of layers: loop forward, then reverse for the backward pass, then update the parameters.
  • A gradient always has the shape of the array it belongs to.

Next lesson: Gradient Checking.