Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 46 ยท Neural Networks for Vision

Pooling and CNN Architectures

Learn max and average pooling, compute receptive fields, trace layer shapes with a helper, and see why residual connections let networks grow very deep.

A convolutional layer keeps the spatial size of an image almost unchanged, and that is expensive: a 224 by 224 image still has 50,000 positions after the first layer. Pooling is the cheap trick that shrinks a feature map by summarizing small neighborhoods. Each time you pool, the map gets smaller, the network sees a larger part of the image per unit, and a tiny shift of the input changes the output less.

In this lesson you will write pooling from scratch, learn the bookkeeping that tells you how much of the image each neuron can "see" (the receptive field), build a helper that traces tensor shapes and parameter counts through a network, and then look at how architectures grew from LeNet to ResNet. If the convolution layer itself is still fuzzy, read The Convolutional Layer first.

Max and average pooling

A pooling layer slides a small window over each channel separately and replaces every window with one number. With a 2 by 2 window and stride 2, the windows do not overlap, so height and width are halved. Max pooling keeps the largest value in the window ("was this feature present anywhere here?"). Average pooling keeps the mean ("how much of this feature is here, overall?"). Pooling has no learnable weights.

The reshape trick below cuts a 6 by 6 array into 2 by 2 blocks without a Python loop: reshape(h//2, 2, w//2, 2) puts the two block axes at positions 1 and 3, so reducing over axis=(1, 3) pools every block at once.

Python Pooling with reshape

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first block holds 1, 3, 4, 2, so max pooling returns 4 and average pooling returns 2.5. Look at the middle row: the 8 survives max pooling untouched, while average pooling dilutes it to 6.0. Max pooling is the default in older CNNs because it behaves like a detector ("is there an edge in this patch?"). Average pooling is common at the very end of a network, where global average pooling collapses each whole feature map to one number.

The slice x[:h - h % size, ...] drops leftover rows or columns when the size is not divisible, which is what most libraries do by default.

Small shifts do not matter much

The main side effect of max pooling is partial translation invariance. If a feature moves by one pixel but stays inside the same window, the pooled output is identical. Move it far enough to cross a window border and the output changes.

Python Pooling absorbs small shifts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Dots A and B sit in columns 2 and 3, which share one window, so their pooled maps match. Dot C is two columns away and lands in the next window. This is why pooling gives only small tolerance to movement, never a complete guarantee. Many modern networks replace pooling with a strided convolution (a convolution that moves two pixels at a time), which downsamples and learns the summary at the same time.

Note: Pooling shrinks the spatial size only. The number of channels stays the same. Networks usually pool while increasing the channel count, so the feature maps get smaller but "deeper" as you go further in.

The receptive field

Every neuron in a CNN looks at a limited patch of the original image. That patch is its receptive field. A neuron in the first 3 by 3 convolution sees 3 by 3 pixels. A neuron in the second sees a 3 by 3 patch of the first layer's outputs, and since neighboring outputs overlap, it covers 5 by 5 pixels of the image. Pooling and strides make this grow faster, because after a stride-2 layer one step in the feature map equals two pixels in the image.

Two numbers per layer are enough to track this: the receptive field rf and the jump j, the distance in input pixels between two neighboring outputs. For a layer with kernel k and stride s:

TEXT
rf_new   = rf + (k - 1) * j
j_new    = j * s
start:   rf = 1, j = 1
flowchart LR
    A["input pixel<br/>rf 1, jump 1"] --> B["conv 3x3<br/>rf 3, jump 1"]
    B --> C["conv 3x3<br/>rf 5, jump 1"]
    C --> D["pool 2x2<br/>rf 6, jump 2"]
    D --> E["conv 3x3<br/>rf 10, jump 2"]

Each layer widens what one neuron sees. After pooling, kernels step through the image two pixels at a time, so later layers widen the field faster.

Python Receptive field calculator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

After seven layers, one neuron looks at a 24 by 24 patch. This is the argument behind the VGG design: three stacked 3 by 3 convolutions see the same 7 by 7 patch as one 7 by 7 convolution, but with 110,592 weights instead of 200,704 (for 64 channels in and out), and with two extra non-linearities between them. Small kernels, stacked deeply, became the standard recipe.

The receptive field also tells you what a network can know. If the object you want to classify is 200 pixels wide and your final receptive field is only 40 pixels, no neuron ever sees the whole object. Real networks have a smaller effective receptive field than the theoretical one, because central pixels contribute more than edge pixels, but the formula is the right first check.

A shape bookkeeping helper

Most of the pain of building CNNs is shape errors. The output size of a convolution or pooling layer along one axis is:

TEXT
out = floor((n + 2 * padding - kernel) / stride) + 1

The helper below walks through a list of layers, prints the output shape (channels, height, width) and counts parameters. A convolution with out_c filters of size k on c input channels has out_c * c * k * k + out_c parameters (weights plus one bias per filter). A dense layer has inputs * outputs + outputs.

Python Tracing shapes through a LeNet-style net

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Study the parameter column. The two convolutions together hold only 2,572 weights, yet the first dense layer after flattening holds 48,120, almost four fifths of the whole network of 61,706 parameters. Convolutions are cheap in parameters but heavy in computation; dense layers are the reverse. That is why later architectures replaced the big dense head with global average pooling.

From LeNet to ResNet

The shape of the network above is almost exactly LeNet (late 1990s), which read handwritten digits on bank cheques: convolution, pooling, convolution, pooling, then dense layers. The decades that followed mostly changed three things: depth, the building block, and training tricks.

flowchart LR
    A["LeNet<br/>1998<br/>few layers"] --> B["AlexNet<br/>2012<br/>ReLU, GPUs, dropout"]
    B --> C["VGG<br/>2014<br/>stacks of 3x3"]
    C --> D["Inception<br/>2014<br/>parallel branches"]
    D --> E["ResNet<br/>2015<br/>skip connections"]

A short family tree of landmark image classifiers. Each step made the network deeper or its blocks smarter.

  • AlexNet showed that a deep CNN trained on GPUs with the ReLU activation could beat hand-crafted features by a wide margin on a large image benchmark.
  • VGG standardized on 3 by 3 convolutions and doubled the channels after each pooling step.
  • Inception ran several kernel sizes side by side inside one block and concatenated the results.
  • ResNet introduced the residual connection and made networks of 50, 101 or 152 layers trainable.

Why residual connections work

Before ResNet, simply adding layers made networks worse beyond a certain depth, even on the training set. Signals and gradients shrank as they passed through many layers. A residual block gives the signal a shortcut: instead of computing y = f(x), it computes y = x + f(x). The layers only have to learn the difference from the input, and the gradient can flow straight through the addition.

flowchart TD
    X["x"] --> F["conv, ReLU, conv<br/>f(x)"]
    X --> ADD(("+"))
    F --> ADD
    ADD --> Y["ReLU(x + f(x))"]

A residual block. The shortcut carries x around the convolutions and adds it back.

You can feel the effect with NumPy alone. The experiment below pushes a vector through 30 layers of small random weights and a ReLU, once as a plain stack and once with the shortcut, and prints the size (norm) of the signal.

Python Plain stack versus residual stack

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

After 30 layers the plain stack has shrunk the signal to about 4e-29: nothing is left to learn from. The residual stack still carries a signal of roughly size 4.7, because the identity path never forgets its input. Real ResNets add batch normalization and learned weights, but the shortcut is the core idea. When the shapes of x and f(x) differ (because of a stride or more channels), the shortcut uses a 1 by 1 convolution to match them.

Tip: You rarely design an architecture from scratch. Pick a known backbone (ResNet, EfficientNet, ConvNeXt), check its input size and receptive field against your objects, and spend your effort on data. The next lessons show how to train a small network and reuse pretrained models.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A 2x2 max pooling layer with stride 2 is applied to a feature map of shape (16, 10, 10). What is the output shape?

    Pooling halves height and width and works on each channel separately, so the 16 channels stay and the 10 by 10 map becomes 5 by 5.
  2. 2. Which statement about max pooling is correct?

    Max pooling has no weights and keeps the strongest response per window. A feature that moves within one window yields the same output, which gives small translation tolerance.
  3. 3. Two stacked 3x3 convolutions with stride 1 are followed by a 2x2 pooling layer with stride 2. What is the receptive field after the pooling layer?

    The two convolutions give 1 + 2 + 2 = 5. The pooling layer adds (2 - 1) * 1 = 1, so the field is 6, and the jump becomes 2.
  4. 4. What does a residual block compute?

    The block adds the input back to the output of its layers. The layers learn a correction to x, and the gradient can flow straight through the addition.
  5. 5. Why did VGG-style networks prefer three 3x3 convolutions over one 7x7 convolution?

    Three stacked 3x3 layers see a 7x7 patch, use fewer parameters (27 C^2 versus 49 C^2) and insert extra ReLU activations between them.

Summary

  • Pooling summarizes small windows (max or average), has no weights, halves the spatial size at stride 2 and keeps the channel count.
  • Max pooling gives tolerance to shifts inside a window; strided convolutions are a learnable alternative.
  • The receptive field grows as rf + (k - 1) * jump, and the jump multiplies by every stride, so pooling widens later fields quickly.
  • Output size along an axis is floor((n + 2p - k) / s) + 1; a shape-tracing helper catches most architecture mistakes early.
  • Dense layers after flattening can hold most of a small CNN's parameters.
  • LeNet, AlexNet, VGG, Inception and ResNet each added depth or better blocks; residual connections (x + f(x)) make very deep networks trainable.

Next up: put these pieces to work and watch a model learn in Training a Small Network on Images.