Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 73 ยท Convolutional Networks

Classic CNN Architectures

A tour of LeNet, AlexNet, VGG, Inception, ResNet and EfficientNet: the ideas behind each design, parameter counts and a residual block in NumPy.

Once you can build a convolutional layer, the next question is how to arrange hundreds of them. For about fifteen years the answer came from a handful of landmark designs, each one fixing a problem of the one before. Knowing them is useful for two reasons: modern vision models still reuse their building blocks, and the problems they solved (too many parameters, vanishing gradients, wasted compute) keep showing up in your own projects.

This lesson walks through LeNet, AlexNet, VGG, Inception, ResNet and EfficientNet. The architectures are far too large to train in a browser, so the real framework code is read-only here, while small NumPy examples let you run the key ideas: counting parameters, the residual connection, and compound scaling.

From digits to ImageNet

LeNet (1998) Conv, pool, conv, pool, dense. Reads digits.
AlexNet (2012) ReLU, dropout and GPUs. Deep learning breaks through.
VGG (2014) Just stack 3x3 convolutions, deeper.
1 / 2

LeNet: the blueprint

LeNet-5 (1998) was built to read handwritten digits on bank cheques. Its layout is the pattern every CNN since has followed: alternate convolution and pooling to shrink the image while adding channels, then finish with dense layers.

TEXT
32x32 input -> conv 5x5 (6 maps) -> pool -> conv 5x5 (16 maps) -> pool
            -> dense 120 -> dense 84 -> dense 10

It used tanh activations and average pooling, and it had about 60,000 parameters. That was small enough to train on 1990s hardware, and it is the same scale as the digit CNN you trained in the previous lesson. Compare it with its descendants:

Network Year Depth Parameters Main idea
LeNet-5 1998 5 about 60 thousand conv, pool, dense
AlexNet 2012 8 about 60 million ReLU, dropout, GPU training
VGG-16 2014 16 about 138 million only 3x3 convolutions
GoogLeNet (Inception v1) 2014 22 about 7 million parallel branches, 1x1 bottlenecks
ResNet-50 2015 50 about 25 million residual connections
EfficientNet-B0 2019 about 80 about 5 million compound scaling

The parameter counts are approximate, but the pattern is the lesson: bigger is not the only road to better accuracy, and ideas about structure eventually beat raw size.

AlexNet: deep learning goes mainstream

In 2012, AlexNet won the ImageNet competition (a million photos in a thousand classes) by a wide margin and started the deep learning boom. It was LeNet's recipe scaled up, with the ingredients that this course has already covered:

  • ReLU activations instead of tanh, which train much faster (Activation Functions).
  • Dropout in the dense layers to fight overfitting (Dropout and Data Augmentation).
  • Data augmentation (random crops, flips) and GPU training, since five convolutional layers on a million images were too slow on CPUs.

Most of its 60 million parameters sit in the last dense layers, not in the convolutions, which is a recurring pattern: convolutions are cheap in parameters, dense layers are not.

VGG: simplicity and depth

VGG asked how far you can get with one rule: use only 3 by 3 convolutions, double the channels after every pooling, and go deep. The justification is a nice calculation. Three stacked 3 by 3 layers see a 7 by 7 region (using the receptive-field rule from Convolution, Pooling and Feature Maps), the same as one 7 by 7 kernel, yet they use fewer weights and add two extra non-linearities. The next TryIt counts them, and also reproduces LeNet's parameter total layer by layer.

Python Counting parameters of LeNet and a VGG block

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

LeNet comes to 61,706 parameters. The stack of three 3 by 3 convolutions needs about 45% fewer parameters than one 7 by 7 kernel for the same field of view. The last line explains why VGG-16 is so large: its first dense layer alone has over 100 million weights, most of the 138 million total. Later designs replaced such dense heads with global average pooling, which removes almost all of them.

VGG's weakness is cost: it needs a lot of memory and computation, and it becomes hard to train beyond 19 layers. The next two ideas address efficiency and depth.

Inception: branches and 1x1 convolutions

Which kernel size is best, 1 by 1, 3 by 3 or 5 by 5? Inception (GoogLeNet) refused to choose: each module runs several branches in parallel on the same input, and concatenates their outputs along the channel axis. The network learns how much weight to give each scale.

Running 5 by 5 convolutions on many channels would be expensive, so the module first squeezes the channel count with a 1 by 1 convolution. A 1 by 1 convolution has no spatial extent; at each pixel it simply takes a weighted combination of all channels, like a tiny dense layer applied per position. It is a cheap way to reduce or mix channels, called a bottleneck. The next example puts a number on the saving.

Python The 1x1 bottleneck saves parameters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two-step version needs only about 26% of the parameters. The same bottleneck trick appears in ResNet-50 and in many later networks.

ResNet: skip connections

By 2015, researchers noticed a puzzle: a 56-layer plain network was worse than a 20-layer one, even on the training data. It was not overfitting; deeper networks were simply harder to optimize, as gradients had to travel through dozens of layers (see Weight Initialization for the signal problem).

ResNet introduced the residual block. Instead of learning a mapping H(x) directly, the block learns the difference F(x) and adds the input back:

TEXT
y = ReLU( F(x) + x )
flowchart LR
    X["x"] --> C1["conv 3x3 + ReLU"]
    C1 --> C2["conv 3x3"]
    C2 --> ADD(("+"))
    X -- "skip connection" --> ADD
    ADD --> R["ReLU"] --> Y["y"]

The input travels past the convolutions on a shortcut and is added to their output.

Two things make this work. First, if the best thing a block can do is nothing, it only has to push F(x) towards zero, which is far easier than learning an identity map through two convolutions. Second, the derivative of F(x) + x with respect to x is F'(x) + 1: the "+ 1" gives gradients a direct highway to earlier layers, no matter how many blocks they cross. This allowed networks with 50, 101 or 152 layers to train well, and ResNet won the 2015 ImageNet competition.

When a block changes the number of channels or halves the size, the shortcut cannot simply copy x; it uses a 1 by 1 convolution (with the same stride) to reshape it. Here is a residual block in NumPy, with the same convolution code as before:

Python A residual block in NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With zero residual weights, the block turns into the identity (followed by the ReLU), which shows how easy it is for a residual network to "skip" a layer that is not needed.

Seeing the gradient highway

The following experiment backpropagates through 30 layers of a plain ReLU stack and through the same stack with skip connections. It uses dense layers to keep the code short, and deliberately small weights, the situation where plain networks struggle. The question is how large the gradient is when it arrives at the first layer.

Python Gradients through 30 layers: plain versus residual

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The plain stack's gradient shrinks by a roughly constant factor per layer, so after 30 layers it is vanishingly small and the early layers would hardly learn. With skip connections the gradient does not vanish at all. Here it even grows, to about 150 times its starting size after 30 layers, because every block adds its branch on top of the straight path; real ResNets keep this in check with normalization and careful initialization. The effect is the same as the "+ 1" in the derivative.

Real ResNets also use batch normalization after each convolution, which helps both effects. Here is the same idea in a framework. This code is read-only (PyTorch is not available in the browser), shown beside what you have already built:

Python
# PyTorch (read-only): a basic ResNet block
import torch.nn as nn
import torch.nn.functional as F

class BasicBlock(nn.Module):
    def __init__(self, c_in, c_out, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(c_in, c_out, 3, stride, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(c_out)
        self.conv2 = nn.Conv2d(c_out, c_out, 3, 1, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(c_out)
        self.shortcut = nn.Sequential()
        if stride != 1 or c_in != c_out:          # projection shortcut
            self.shortcut = nn.Sequential(
                nn.Conv2d(c_in, c_out, 1, stride, bias=False), nn.BatchNorm2d(c_out))

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        return F.relu(out + self.shortcut(x))

EfficientNet: scaling the right way

If you have more compute, how should you grow a network? You can make it deeper (more layers), wider (more channels) or feed it higher-resolution images. Earlier practice picked one, often depth. EfficientNet observed that the three work best together, and proposed compound scaling: choose fixed ratios alpha (depth), beta (width) and gamma (resolution), and scale all three using a single knob phi:

TEXT
depth x alpha^phi    width x beta^phi    resolution x gamma^phi

The cost of a conv network grows linearly with depth but with the square of width and of resolution, so the ratios are chosen such that alpha * beta^2 * gamma^2 โ‰ˆ 2. Each step of phi then roughly doubles the compute. The published values are about 1.2, 1.1 and 1.15.

Python Compound scaling arithmetic

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At phi = 3 compute grows about 7 times, and depth alone would need roughly 7 times more layers to spend the same budget, which is exactly the lopsided scaling that EfficientNet avoids. The EfficientNet family (B0 to B7) also uses inverted bottleneck blocks and squeeze-and-excitation, ideas you can look up once you are comfortable with the basics.

MobileNet: convolutions that fit on a phone

VGG and ResNet were built for servers. MobileNet (2017) was designed for phones, cameras and microcontrollers, where memory and battery are the limits. Its single big idea is the depthwise separable convolution, which splits one expensive operation into two cheap ones:

  1. A depthwise convolution applies one k x k filter to each input channel on its own. Channels never mix, and there are only C_in * k * k weights.
  2. A pointwise convolution (1 x 1) then mixes the channels, with C_out * C_in weights.

A standard convolution needs C_out * C_in * k * k weights, and the same ratio holds for multiply-adds. The separable version costs about 1/C_out + 1/k^2 of that, which is roughly 8 to 9 times less for 3 by 3 kernels. The price is a small loss of accuracy, because each filter is restricted to a special low-rank form.

Python Depthwise separable convolution: same shape, a fraction of the cost

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two routes give identical shapes. For 32 input and 64 output channels the standard layer needs 18,432 weights and about 3.6 million multiply-adds, while the separable pair needs 2,336 weights and 0.46 million, a ratio of 0.127. The check at the end shows that a separable convolution is just a standard convolution whose filters are built from the product of a depthwise and a pointwise part. MobileNet adds a width multiplier alpha that scales every channel count, so one design gives a whole range of sizes: at alpha = 0.5 this block has 656 weights, because both channel counts are halved.

MobileNetV2 wraps the idea in an inverted residual block: a 1 by 1 convolution first expands the channels (by a factor of 6), a depthwise 3 by 3 convolution works in that wide space, and a 1 by 1 convolution projects back to a narrow layer, with a skip connection joining the narrow ends. It is the reverse of the ResNet bottleneck, which squeezes in the middle. The final narrow layer has no ReLU, since a ReLU would destroy information in a low-dimensional space. EfficientNet reuses these blocks. Model Compression covers other ways to shrink a trained network.

Vision Transformers (ViT)

Convolutions build in two assumptions: pixels close together matter most, and the same pattern means the same thing anywhere. The Vision Transformer (ViT, 2020) drops both and lets attention learn what matters, using the very same transformer block as language models. The only new piece is how to turn a picture into a sequence of tokens:

  1. Cut the image into a grid of non-overlapping patches, for example 16 by 16 pixels each.
  2. Flatten every patch and pass it through one shared linear layer. This is the patch embedding. It is mathematically a convolution whose kernel size equals its stride.
  3. Prepend a learnable [CLS] token whose final state is used for classification.
  4. Add position embeddings, because attention alone does not know where a patch came from (see Self-Attention and Multi-Head Attention).

The sequence then flows through the usual stack of attention and feed-forward blocks, and a small classifier reads the [CLS] output.

Python From an image to a token sequence

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A 32 by 32 image with 4 by 4 patches becomes 64 tokens plus the [CLS] token. The embedding really is a strided convolution (the check prints True), and one attention layer lets every patch see every other patch, while a 3 by 3 convolution reaches just 9 pixels per layer. The cost is the last table: the attention matrix grows with the square of the token count, so halving the patch size from 16 to 8 on a 224 by 224 image multiplies it by 16.

ViT has little built-in image knowledge, so on small datasets a ResNet usually wins. With very large data (hundreds of millions of images) or heavy pretraining and augmentation, ViT matches or beats convolutional networks, and it is the backbone of most modern image-text models. Variants such as the Swin Transformer limit attention to local windows to cut the cost, and hybrid models (ConvNeXt and others) bring convolutional habits back into transformer-style designs.

U-Net in a nutshell

You met U-Net in the segmentation lesson. In architecture terms it is the one network in this list that outputs an image the same size as its input: an encoder that halves the resolution and doubles the channels at each level, a decoder that does the reverse with upsampling or transposed convolutions, and skip connections that concatenate each encoder map into the decoder level of the same size. Those skips are what make fine boundaries sharp. The same shape sits inside diffusion models, where a U-Net predicts the noise at every pixel (see Generative Models).

Which one should you use?

For a new image problem you will almost never design an architecture yourself. You pick a standard one such as ResNet-50 or an EfficientNet, start from weights trained on ImageNet, and adapt it. That is the topic of Transfer Learning and Fine-Tuning. Newer families, like ConvNeXt, build on exactly the ideas above (stacked blocks, residual connections, normalization and careful scaling), and for phones MobileNet or EfficientNet-Lite are the usual starting points.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why did VGG use stacks of 3x3 convolutions instead of larger kernels?

    Three 3x3 layers see a 7x7 region like one 7x7 kernel, but with about 45% fewer weights for equal channel counts, and with two extra ReLUs.
  2. 2. What does a residual block compute?

    The skip connection adds the unchanged input to the convolution branch, so the branch learns only a residual correction.
  3. 3. What is the main job of a 1x1 convolution in an Inception module or a bottleneck block?

    A 1x1 convolution combines channels at each pixel. Reducing channels first makes the following 3x3 or 5x5 convolution much cheaper.
  4. 4. What is the central claim of EfficientNet's compound scaling?

    Scaling one dimension saturates quickly. Growing all three in balance, set by one coefficient phi, uses compute more effectively.

Summary

  • LeNet set the blueprint: alternate convolution and pooling, then dense layers. AlexNet scaled it up with ReLU, dropout and GPUs.
  • VGG stacks 3 by 3 convolutions, which cover large fields with fewer weights; its dense head holds most of its 138 million parameters.
  • Inception runs parallel branches and uses 1 by 1 convolutions as cheap channel bottlenecks.
  • ResNet adds the input back to a block's output, y = ReLU(F(x) + x), giving gradients a direct path and making very deep networks trainable.
  • A projection shortcut (1 by 1 convolution) is used when a block changes the channel count or size.
  • EfficientNet scales depth, width and resolution together, since compute grows with depth times the squares of width and resolution.
  • MobileNet replaces a convolution with a depthwise plus a pointwise (1 by 1) one, cutting cost to about 1/C_out + 1/k^2; V2 adds inverted residual blocks.
  • A Vision Transformer cuts the image into patches, embeds them (a strided convolution), adds [CLS] and position embeddings, and runs a transformer; attention cost grows with the square of the token count.
  • U-Net is the encoder-decoder with skip connections used when the output must be an image.

Next: Transfer Learning and Fine-Tuning.