Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 73 ยท Convolutional Networks

Training a CNN From Scratch

Derive the backward pass of convolution and max pooling, implement it with im2col in NumPy, and train a small CNN on 8x8 handwritten digits.

A convolutional network learns exactly like the networks you built earlier: a forward pass computes predictions and a loss, a backward pass computes the gradient of the loss with respect to every weight, and an optimizer takes a step. Only two layers are new, convolution and max pooling, and for each you need to know how gradients flow backward through it.

The trick that makes this manageable is im2col: rewrite the convolution as an ordinary matrix multiplication by unrolling every patch of the image into a row. Once a convolution is a matrix product, its backward pass is the dense layer backward pass you already know, followed by a small step that puts the patch gradients back into image positions. At the end of this lesson you will train a complete CNN on handwritten digits using nothing but NumPy.

Backward through a conv layer

im2col Unroll every patch into one row of a big matrix.
One matmul Patches times filters gives every output at once.
Reuse dense backprop The same two matrix products give dW and the patch gradients.
col2im Add patch gradients back to the pixels they came from.
1 / 2

Convolution as a matrix product

Last lesson's conv_forward used einsum. For training we spell the same thing out. With im2col each position of the sliding window becomes one row, and the row holds all C * kh * kw numbers the filter would multiply. For an input (N, C, H, W) and stride 1 the matrix cols has shape (N * out_h * out_w, C * kh * kw). The filters (F, C, kh, kw) are reshaped to (F, C * kh * kw). Then:

TEXT
out = cols @ W.T + b          # shape (N * out_h * out_w, F)

Reshape the result to (N, out_h, out_w, F) and move F forward, and you have the feature maps. This is a plain linear layer whose "examples" are patches.

That is also why the gradients are familiar. With d the incoming gradient reshaped to (N * out_h * out_w, F):

  • dW = d.T @ cols, then reshaped back to (F, C, kh, kw). Every patch contributes, so the gradient of a shared weight is summed over all positions.
  • db = d.sum(axis=0).
  • dcols = d @ W gives the gradient for every patch. A pixel belongs to up to kh * kw overlapping patches, so its gradient is the sum of what each patch reports. The reverse of im2col, col2im, performs that accumulation.

The next example implements all three pieces and verifies them with finite differences, the technique from Gradient Checking.

Python Conv forward and backward with im2col

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All three relative errors are tiny, so the im2col forward and backward passes are correct. Note how short the backward pass is: two matrix products and one accumulation loop that runs only kh * kw times, not once per pixel.

Backward through max pooling

Max pooling copies the largest value of each window into the output. Therefore, a small change in the winning input changes the output one-for-one, while a change in any other input changes nothing. The gradient follows the same rule: the whole upstream gradient goes to the position that won, and the other positions in the window get zero.

In NumPy: build a mask that marks the winners, and multiply it by the upstream gradient. If two entries tie for the maximum, we split the gradient between them, which keeps the total correct.

Python Max pooling backward: routing the gradient

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In the top-left window (1, 3, 4, 2) the 4 wins, so it receives all 10. The top-right window (2, 0, 1, 1) sends 20 to its 2, and the bottom-right window sends 40 to the 7. The bottom-left window (0, 1, 2, 2) has a tie, so the two 2s each get half of 30, which is 15. The total of 100 is preserved.

ReLU and the flatten step have simple backward passes too. ReLU passes the gradient where its input was positive, and flatten just reshapes the gradient back.

A small CNN, end to end

Now assemble the network:

TEXT
input (1, 8, 8) -> conv 3x3, 8 filters, padding 1 -> ReLU -> max pool 2
                -> flatten (8 x 4 x 4 = 128) -> dense 10 -> softmax cross-entropy

The data is scikit-learn's built-in load_digits: 1,797 handwritten digits as 8 by 8 grayscale images, scaled here to the range 0 to 1. We keep 1,400 for training and the rest for testing, and train with the Adam optimizer from Momentum, RMSProp and Adam. Each layer is a small class with forward and backward, the same design you will use in the mini framework.

Python Training a CNN on 8x8 digits

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The network has just 1,370 trainable parameters. Test accuracy climbs from about 49% after the first epoch to about 97% after twenty. The third panel shows the eight learned 3 by 3 filters. Nobody designed them: gradient descent shaped them from random starting values, and learned filters often look like rough stroke and edge detectors, much like the hand-made ones in the previous lesson.

Speed notes

Training a CNN in pure NumPy is slow compared with a framework, and the in-browser Python runner stops after 30 seconds, so a few habits matter:

  • Never loop over pixels in Python. The loops that remain here run over the kernel offsets (9 iterations), not over the image. Replacing a per-pixel loop with im2col is typically a speed-up of tens of times.
  • The first layer needs no input gradient. Passing need_dx=False skips a whole col2im where nobody would use the result.
  • Keep the images small and the filter count modest. The cost of a conv layer grows with N * out_h * out_w * C * kh * kw * F. An 8 by 8 image with 8 filters is trivial; a 224 by 224 color image with 64 filters is roughly 20,000 times more work per image.
  • Use float32 for real training. NumPy's default is float64, which is twice the memory traffic. Frameworks also run the same matrix products on GPUs.
Python Why im2col beats loops

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The cost of im2col is memory: every pixel is copied into up to kh * kw rows. For 3 by 3 kernels the matrix is about nine times larger than the input. Production libraries use cleverer algorithms (direct convolution, Winograd, FFT) that avoid the copy, but the gradient formulas are identical.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. After im2col, a convolution layer's forward pass is:

    Each row of cols is one patch and each filter is one column of W.T, so cols @ W.T computes every output value in one product.
  2. 2. Why is the gradient of a conv filter's weights summed over all positions?

    A shared weight affects every output cell it was applied to. By the chain rule its total gradient is the sum of its contributions from all those cells.
  3. 3. In the backward pass of 2x2 max pooling with no ties, where does the upstream gradient of a window go?

    Only the winning input influenced the output, so only it receives gradient.
  4. 4. What does col2im do in the backward pass?

    Neighboring patches overlap, so a pixel appears in several rows of cols. Its gradient is the sum over all of those rows.

Summary

  • im2col turns a convolution into one matrix product, so conv backward reuses the dense-layer formulas: dW = d.T @ cols, db = d.sum(0) and dcols = d @ W.
  • col2im adds the patch gradients back into image positions, summing where patches overlap.
  • Max pooling sends the upstream gradient only to the winning element of each window (split equally on ties).
  • ReLU masks the gradient where its input was not positive; flatten only reshapes it.
  • Finite-difference checks confirm each hand-written backward pass.
  • With im2col, need_dx=False for the first layer and small images, a CNN on 8 by 8 digits trains in seconds even in the browser.

Next: Classic CNN Architectures.