Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 45 ยท Computer Vision

CNN Architectures, Pooling and ResNet

Tour the landmark CNN designs from LeNet to ResNet, understand pooling types and residual connections, and see why skip connections fix the degradation problem.

Once you know what a convolution does (Convolutions and CNNs), the next question is how to arrange many of them. A few landmark architectures set the patterns that almost every vision model still follows. This lesson walks through them in order, looks closely at pooling, and then explains the idea that made very deep networks trainable: the residual connection of ResNet.

PyTorch cannot run in the browser, so PyTorch code is read-only. The live editors are NumPy stand-ins that compute the same quantities: shapes, parameter counts, pooling results and how gradients behave in deep stacks.

Pooling in depth

A pooling layer summarizes each small window of a feature map with one number. It has no learnable weights, shrinks the spatial size, and makes the network a little tolerant to small shifts of the input.

Layer Computes Typical use
nn.MaxPool2d(2) Largest value in each window Keeps the strongest feature response; the classic choice
nn.AvgPool2d(2) Mean of each window Smoother summary; used inside some blocks
nn.AdaptiveAvgPool2d(1) Mean of each whole channel (global average pooling) Final step before the classifier; works for any input size
nn.AdaptiveMaxPool2d(1) Maximum of each whole channel (global max pooling) Detects "was this feature present anywhere?"
Python
import torch.nn as nn

nn.MaxPool2d(kernel_size=2, stride=2)      # 28x28 -> 14x14
nn.AvgPool2d(kernel_size=2, stride=2)
nn.AdaptiveAvgPool2d((1, 1))               # (N, C, H, W) -> (N, C, 1, 1), any H and W

Advantages: fewer values for later layers (less compute and memory), a larger receptive field, a little translation invariance, no parameters to overfit. Limitations: pooling throws information away, including exact positions, which hurts tasks such as segmentation; many modern networks replace pooling in the body with a convolution of stride 2 and keep only global average pooling at the end.

Python Max, average and global pooling by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The landmark architectures

LeNet-5 (1998)

Yann LeCun's network read handwritten digits on bank checks. It stacks two convolution and pooling stages and then dense layers: conv(5x5) - pool - conv(5x5) - pool - 120 - 84 - 10, about 60,000 parameters. It set the template: convolutions extract features, pooling shrinks them, dense layers classify.

AlexNet (2012)

AlexNet won ImageNet by a wide margin and started the deep learning boom. It was deeper (five convolutional and three dense layers, about 60 million parameters), used ReLU instead of tanh for faster training, dropout in the dense layers, data augmentation, and trained on two GPUs.

VGG (2014)

VGG showed that depth with small 3 by 3 kernels works: blocks of two or three 3x3 convolutions followed by max pooling, repeated, giving 16 or 19 weight layers. It is simple and uniform, but heavy: VGG-16 has about 138 million parameters, mostly in its dense layers.

GoogLeNet and Inception (2014)

GoogLeNet used Inception modules: parallel branches with 1x1, 3x3 and 5x5 convolutions and pooling, concatenated along the channel axis, so the network picks the best scale itself. 1x1 convolutions reduce the number of channels before the expensive kernels, cutting compute. It had only about 7 million parameters, thanks to global average pooling instead of huge dense layers.

ResNet (2015)

ResNet made networks with 50, 101 or 152 layers train reliably. The idea is explained next.

Network Year Depth Parameters Key idea
LeNet-5 1998 5 ~60 K Conv, pool, dense template
AlexNet 2012 8 ~60 M ReLU, dropout, GPUs
VGG-16 2014 16 ~138 M Stacked 3x3 convolutions
GoogLeNet 2014 22 ~7 M Inception modules, 1x1 convs
ResNet-50 2015 50 ~25 M Residual connections
Python Output shapes and parameters of a LeNet-style network

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The formula (size + 2*padding - kernel) // stride + 1 is the same one used in Convolutions and CNNs.

The degradation problem

Depth should help: a deeper network can in principle copy a shallower one by making the extra layers do nothing. In practice, plain networks beyond about 20 layers got worse, even on the training set. This is called the degradation problem, and it is not overfitting (training error rises too). Two issues combine: gradients shrink or blow up as they pass through many layers (the vanishing or exploding gradient problem, partly tamed by good initialization and normalization), and, even with those, plain stacks find it hard to learn the identity function.

Residual connections

ResNet's fix is to let a block learn only a correction to its input. A residual block computes

Text
output = F(x) + x

where F is a small stack of layers (two 3x3 convolutions with batch normalization) and x is carried around them by a skip connection. If the best thing a block can do is nothing, it only has to push F toward zero, which is easy.

flowchart LR
    X["x"] --> C1["conv 3x3<br/>BN, ReLU"]
    C1 --> C2["conv 3x3<br/>BN"]
    C2 --> ADD(("+"))
    X -->|"skip connection"| ADD
    ADD --> R["ReLU"]

A basic residual block: the input is added back to the output of two convolutions.

The skip connection also gives gradients a short, unobstructed path to early layers: the derivative of F(x) + x with respect to x is F'(x) + 1, so the gradient always has the "+1" term. The block in PyTorch is short:

Python
import torch.nn as nn
import torch.nn.functional as F

class BasicBlock(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.conv1 = nn.Conv2d(channels, channels, 3, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(channels)
        self.conv2 = nn.Conv2d(channels, channels, 3, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(channels)

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        return F.relu(out + x)          # the skip connection

When a block changes the number of channels or the spatial size (stride 2), x no longer has the right shape, so the skip path uses a projection shortcut: a 1x1 convolution with the same stride that reshapes x before the addition.

Python Gradients through 50 layers: plain versus residual

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The plain stack's gradient shrinks geometrically with depth, while the residual one stays usable (it can even grow, which is why careful initialization and normalization still matter).

ResNet variants

ResNet-18 and ResNet-34 stack basic blocks in four stages with 64, 128, 256 and 512 channels; each stage after the first halves the spatial size with a stride-2 block. ResNet-50, 101 and 152 use bottleneck blocks (1x1 reduce, 3x3, 1x1 expand) to keep compute down as depth grows. The network starts with a 7x7 stride-2 convolution and a max pool, and ends with global average pooling and one dense layer. Torchvision provides all of them, pretrained:

Python
from torchvision import models

resnet = models.resnet50(weights=models.ResNet50_Weights.DEFAULT)
print(sum(p.numel() for p in resnet.parameters()))   # about 25.6 million

Advantages: very deep networks train, accuracy improves with depth, and the design is simple and reusable (it is the backbone behind many detection and segmentation models; see Transfer Learning). Challenges: more layers cost more compute and memory, very deep versions can still overfit small datasets, and the skip connections need matching shapes (hence projections). Residual connections later became standard in transformers too.

Tip: For a new vision task, start from a pretrained ResNet-18 or ResNet-50 before designing anything yourself. It is a strong, cheap baseline.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which layer has no learnable parameters?

    Pooling only summarizes windows. Convolution, linear and batch norm layers all have weights.
  2. 2. What does global average pooling do?

    It reduces (N, C, H, W) to (N, C, 1, 1), so the classifier does not depend on the input resolution.
  3. 3. What is the degradation problem?

    Beyond some depth, plain stacks became harder to optimize, so even training accuracy dropped.
  4. 4. A residual block computes...

    The block adds its input back to the transformed output, so it only has to learn a correction.
  5. 5. Why does a projection shortcut (1x1 convolution) sometimes appear on the skip path?

    The two terms of F(x) + x must have identical shapes. A 1x1 convolution with a stride reshapes x when the block changes them.

Summary

  • Pooling summarizes windows without weights; max pooling keeps strong responses, average pooling smooths, global pooling replaces huge dense layers.
  • LeNet set the conv-pool-dense template; AlexNet added ReLU, dropout and GPUs; VGG stacked 3x3 kernels; GoogLeNet used Inception modules and 1x1 convolutions.
  • Plain networks beyond a certain depth degrade; residual blocks compute F(x) + x so layers learn corrections and gradients flow through the skip path.
  • ResNet-18/34 use basic blocks, ResNet-50+ use bottleneck blocks; torchvision offers all of them pretrained.

Next: classify real images end to end in Image Classification.