Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 46 ยท Neural Networks for Vision

The Convolutional Layer

How a convolutional layer works: learned kernels, feature maps, channels, output size, parameter counts versus dense layers, and a forward pass in NumPy.

In the earlier chapters you wrote the filters yourself: a blur kernel, a Sobel kernel, a Gaussian. You chose the numbers because you knew what you wanted, such as smoothing or edges. A convolutional layer keeps the same sliding-window mechanism but makes the numbers learnable. Instead of deciding that a vertical edge detector looks like [-1, 0, 1], the network starts with random kernels and adjusts them until they help solve the task, whether that is recognizing digits, cats or tumors.

The convolutional layer is the basic building block of the networks that transformed computer vision. Two ideas make it far more efficient than the obvious alternative of connecting every pixel to every unit: each output looks only at a small neighborhood, and the same weights are reused at every position. This lesson builds the layer in NumPy, counts its parameters, shows what weight sharing buys, and lets a kernel learn a filter from examples. If sliding windows are new to you, read Convolution and Correlation first.

Anatomy of the layer

A convolutional layer takes a stack of 2D maps called channels and produces another stack of channels. A color photo has 3 input channels (R, G, B). The layer has several filters, and each filter produces one output channel, called a feature map.

For a single filter, the computation at each position is:

  1. Place a small window (the kernel size, commonly 3 by 3) on the input.
  2. Multiply the window by the filter's weights across all input channels and add everything up.
  3. Add one bias number.
  4. Write the result to the feature map at that position.

So a filter is not a flat 3 by 3 grid but a small block of shape (C_in, kh, kw), with one 3 by 3 slice per input channel. The whole layer's weights are an array of shape (C_out, C_in, kh, kw). Normally a nonlinearity such as ReLU (max(0, x)) follows, turning negative responses into zero; without it, stacked layers would collapse into a single linear filter.

flowchart LR
    A["Input<br/>(C_in, H, W)"] --> B["C_out filters<br/>each (C_in, kh, kw)"]
    B --> C["Slide, multiply,<br/>sum, add bias"]
    C --> D["Feature maps<br/>(C_out, H_out, W_out)"]
    D --> E["ReLU<br/>max(0, x)"]

One layer turns C_in input channels into C_out feature maps, one per filter.

Output size

Two settings control the spatial size of the output:

  • Padding adds a border of zeros around the input. With a 3 by 3 kernel, pad=1 lets the window center on border pixels, so the output keeps the input size ("same" convolution).
  • Stride is how far the window jumps. A stride of 2 skips every other position and halves the output size.

The formula for one dimension is:

out = (n + 2 * pad - k) // stride + 1

where n is the input size and k the kernel size. For n = 8, k = 3, pad = 1, stride = 1 it gives 8; with pad = 0 and stride = 2 it gives 3.

A forward pass in NumPy

The function below is a direct, slow, readable version of what libraries do with optimized code. It accepts an input of shape (C_in, H, W) and weights of shape (C_out, C_in, kh, kw). We give it two hand-made kernels, a vertical-edge detector and a horizontal-edge detector, so you can see what feature maps look like. In a trained network the numbers would be learned.

Python A convolutional layer from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two filters produce two feature maps, so the output shape is (2, 8, 8). The vertical-edge map lights up in columns 3 and 4, on both sides of the boundary between the dark and bright halves of the image, with the maximum possible value of 4 where the edge is clean. The horizontal-edge map responds just above the bright bar (row 5), because brightness increases going down there. Below the bar the response is negative, and ReLU turns it into 0, so one filter reports only one direction of change.

Look also at the other nonzero values. The ones along the top row of the horizontal map and the bottom left of the vertical map are not real edges of the image: they come from the zero padding, which looks like a dark frame around the picture. Padding is convenient but is not free of artifacts at the border.

With stride 2 and no padding the same layer shrinks the output to (2, 3, 3), matching the formula.

Note: Deep learning libraries call this operation "convolution" although they compute cross-correlation: the kernel is not flipped. Since the kernel is learned, the flip makes no difference to what the network can represent. See Convolution and Correlation for the distinction.

Kernels that are learned

How does a kernel find good numbers? By gradient descent: compare the layer's output with what the task wants, compute how each weight should change to reduce the error, and nudge the weights. The same process trains every layer of a network. Full network training comes in Training a Small Network on Images; here is the principle on one 3 by 3 kernel.

We create 40 pictures with random bright rectangles. The "teacher" signal is the output of the Sobel vertical-edge kernel. A learner that starts from small random weights has to rediscover this kernel from the examples alone.

Python A kernel learns an edge detector

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss falls from about 2 to 0.0002 and the learned weights approach the target: roughly -1, 0, 1 in the outer rows and -2, 0, 2 in the middle row. Notice how the code works. Every 3 by 3 window of every image is flattened into a row of patches, and the kernel is one weight vector of length 9 that is applied to all of them. The gradient is the average over all positions, so one set of weights learns from every location at once. A real network learns thousands of such kernels together, each from the task's loss, without being told what an edge is. Visualizing the first-layer kernels of trained image networks regularly shows edge and color-blob detectors like this one.

Why not a dense layer?

The obvious way to process an image is a dense (fully connected) layer: flatten the image and connect every input number to every output unit. It works for tiny inputs, but it scales badly. A 3 by 3 convolution with 16 filters on a 32 by 32 RGB image is compared with a dense layer producing the same 32 by 32 by 16 output:

Python Counting parameters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The dense layer needs 50,348,032 parameters, while the convolutional layer needs 448. That is 16 * 3 * 3 * 3 = 432 weights plus 16 biases. The convolutional count depends only on the number of filters, the number of input channels and the kernel size. It does not depend on the image size, while the dense count explodes: nearly 13 billion for a 128 by 128 image.

The rule for a convolutional layer is C_out * C_in * kh * kw + C_out. Notice the last line of the output: a 1 by 1 kernel with 64 input channels and 64 filters has 4,160 parameters. A 1 by 1 convolution does not look at neighbors; it mixes channels at each pixel, and it is a common tool for changing the number of channels cheaply.

Weight sharing and translation equivariance

The saving comes from two design choices.

  • Local connectivity: each output depends only on a small window, because nearby pixels are related and far-away ones usually are not.
  • Weight sharing: the same kernel is used at every position, because a pattern such as an edge or a corner means the same thing wherever it appears.

Weight sharing has a precise consequence called translation equivariance: if you shift the input, the feature map shifts by the same amount. "Equivariant" means the output moves along with the input. The next example moves an object 2 pixels down and 3 pixels right and checks the layer's response.

Python Shift the input, the output shifts too

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The check conv(shift(x)) == shift(conv(x)) prints True: the strongest response moves from position (1, 4) to (3, 7), exactly (+2, +3), with the same value 2.251. A dense layer has no such property. Its outputs for the shifted image differ from the original's by 4.187 on average, because it must learn about each position separately.

Equivariance is not the same as invariance. Invariance means the output does not change when the input moves, which is what a classifier wants ("a cat is a cat wherever it sits"). Convolution alone preserves the position information in the feature map. Invariance comes from later steps that discard position, such as the global maximum in the last line (2.251 both before and after) or the pooling layers of the next lesson. The digits experiment in Classifying Handwritten Digits showed what happens without these properties: pixel-based classifiers broke when digits moved by one pixel.

flowchart TB
    A["Same kernel at every position"] --> B["Few parameters<br/>independent of image size"]
    A --> C["Translation equivariance<br/>shift in, shift out"]
    C --> D["Pooling or global max<br/>discards position"]
    D --> E["Translation invariance<br/>for classification"]

Weight sharing gives both efficiency and equivariance; invariance is added by discarding position later.

Stacking layers and what comes next

One layer looks at 3 by 3 windows. Stack a second 3 by 3 layer on top and each output depends on a 5 by 5 region of the original image: this is the receptive field, and it grows with depth. Early layers tend to find edges and color blobs, middle layers combine them into textures and parts, and late layers respond to whole objects. Between layers, networks insert pooling or strides to shrink the maps, and finish with a small dense layer that produces class scores. That architecture is the subject of the next lesson, Pooling and CNN Architectures.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A convolutional layer has 32 filters, takes 16 input channels and uses 3 x 3 kernels. How many parameters does it have (weights plus biases)?

    Weights: 32 * 16 * 3 * 3 = 4,608. Biases: one per filter, 32. Total 4,640.
  2. 2. An input has size 32. With kernel 5, padding 2 and stride 1, what is the output size?

    (32 + 2*2 - 5) // 1 + 1 = 32. Padding of (k - 1) / 2 keeps the size for odd kernels with stride 1.
  3. 3. What does 'translation equivariance' of a convolutional layer mean?

    Because the same kernel is applied at every position, moving a pattern moves its response. Invariance (no change at all) needs an extra step such as pooling.
  4. 4. Why does a convolutional layer need far fewer parameters than a dense layer on the same image?

    Parameter count depends on filters, channels and kernel size, not on image size. A dense layer needs a weight for every input-output pair.
  5. 5. What is the purpose of the ReLU after a convolution?

    Without a nonlinearity, a stack of convolutions is equivalent to one convolution. ReLU keeps positive responses and sets negative ones to zero.

Summary

  • A convolutional layer slides learnable kernels over a multi-channel input. Weights have shape (C_out, C_in, kh, kw); each filter yields one feature map, usually followed by ReLU.
  • Output size per dimension is (n + 2*pad - k) // stride + 1; padding keeps size, stride shrinks it. Zero padding can create artificial responses at the border.
  • Kernels are learned by gradient descent from the task's loss; a single kernel trained on examples rediscovers a Sobel edge filter.
  • Parameters: C_out * C_in * kh * kw + C_out, independent of image size. The example layer needs 448 parameters where a dense layer needs about 50 million.
  • Weight sharing gives translation equivariance (shift in, shift out); invariance is added by pooling or global maximum.
  • Stacking layers grows the receptive field, from edges toward objects.

Next: Pooling and CNN Architectures.