Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 46 ยท Neural Networks for Vision

Classic CNN Architectures: LeNet, VGG and DenseNet

Trace the history from LeNet to VGG and DenseNet: count layers and parameters in Python, see the design rules behind VGG-16 and VGG-19, and how dense blocks work.

Every modern vision network descends from a handful of designs that each made one big idea clear. LeNet showed that convolutions plus pooling can read digits. AlexNet showed that the same idea wins on large images with GPUs. VGG showed that a very regular stack of small 3 by 3 filters works better than clever irregular layers. ResNet added shortcuts so networks could be hundreds of layers deep, and DenseNet took shortcuts to their extreme by connecting every layer to every later layer in a block.

You will rebuild three of them as shape and parameter calculators in plain Python, so you can see where the numbers come from, and learn the design principles behind each. This builds on The Convolutional Layer and Pooling and CNN Architectures (which introduced the shape-tracing helper and residual connections).

A short history

Year Network What it contributed
Late 1980s Early convnets (Fukushima's Neocognitron, LeCun's first digit readers) Local receptive fields, weight sharing, trained with backpropagation
1989 LeNet-1 A convolutional network that read handwritten zip codes for the US Postal Service
1998 LeNet-5 The classic form: conv, pool, conv, pool, then dense layers; about 60,000 parameters
2000s Quiet years Hand-made features (SIFT, HOG) and SVMs dominated; data and computers were too small
2012 AlexNet ReLU, dropout and GPUs, large data (ImageNet); cut the error rate sharply
2014 VGG, GoogLeNet Depth of 16 to 22 layers; very uniform 3x3 design, and Inception modules
2015 ResNet Residual connections: 50 to 152 layers
2017 DenseNet Dense connectivity with feature reuse
2020 onward Vision transformers, ConvNeXt, EfficientNet Scaling rules, attention, modernised convolutions

LeNet's invention was not only the architecture but the motivation. Fully connected networks on raw pixels need a different weight for every pixel pair and do not respect that nearby pixels are related or that the same pattern can appear anywhere. Convolution fixes both problems at once, with far fewer parameters.

LeNet-5 layer by layer

LeNet-5 reads 32 by 32 grayscale digits. Its layers are:

  1. C1: convolution, 6 filters of 5x5, giving 6 maps of 28x28
  2. S2: subsampling (pooling) 2x2, giving 6 maps of 14x14
  3. C3: convolution, 16 filters of 5x5, giving 16 maps of 10x10
  4. S4: pooling 2x2, giving 16 maps of 5x5
  5. C5: convolution 5x5 that acts as a fully connected layer with 120 units
  6. F6: dense layer with 84 units
  7. Output: 10 classes
Python Trace LeNet-5 shapes and parameters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

About 61,700 parameters, and almost four fifths of them sit in the dense layer after flattening (48,120 for 400 -> 120). The original LeNet-5 used a hand-chosen sparse connection table in C3, so exact numbers differ slightly between implementations, and modern versions use ReLU and max pooling in place of tanh and subsampling. Notice the pattern that every CNN since has inherited: spatial size shrinks, channel count grows.

VGG: depth through repetition

The VGG team at Oxford (2014) asked a clean question: what happens if we keep the design as simple as possible and just go deeper? Their rules:

  • Use only 3x3 convolutions with stride 1 and padding 1, so the size never changes inside a block.
  • After each block of convolutions, halve the size with 2x2 max pooling and double the channels (64, 128, 256, 512, 512).
  • Finish with three dense layers (4096, 4096, 1000).

Why 3x3 only? Two stacked 3x3 convolutions see a 5x5 area, three see 7x7, but cost fewer parameters than one big filter (3 x 3^2 = 27 against 7^2 = 49 weights per channel pair) and add more non-linearities. Deep stacks of small filters win.

VGG-16 has 13 convolutional layers and 3 dense layers (16 weight layers). VGG-19 adds one convolution to each of the last three blocks (16 convolutional + 3 dense = 19 weight layers). Configuration, as numbers of 3x3 convolutions per block:

Block Channels VGG-16 VGG-19
1 64 2 2
2 128 2 2
3 256 3 4
4 512 3 4
5 512 3 4
Python Count VGG-16 and VGG-19 parameters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The totals are the published figures: about 138 million for VGG-16 and 144 million for VGG-19. Look at where they sit: the first dense layer alone (25,088 x 4096) holds more than 100 million weights, which is why later designs (ResNet, DenseNet) replaced the big dense head with global average pooling. VGG's drawbacks are exactly this size and its slow, memory-hungry computation, but its simple uniform structure makes it a good teaching network and a favourite backbone for feature extraction and style transfer.

VGG-16 is also used with extra heads for object localization: besides the 1000 class scores, a branch predicts four box numbers (either one box shared by all classes or one class-specific box per class), trained with a regression loss. That is the starting point of the detectors in Object Detection Basics.

DenseNet: every layer sees everything before it

ResNet adds the input of a block to its output (x + f(x)). DenseNet (Huang et al., 2017) instead concatenates: each layer receives the feature maps of all earlier layers in its block and passes its own forward to all later ones. A block with L layers has L(L+1)/2 connections instead of L.

graph LR
    X0["x0 (input)"] --> L1["layer 1"]
    X0 --> L2["layer 2"]
    X0 --> L3["layer 3"]
    L1 --> L2
    L1 --> L3
    L2 --> L3
    L3 --> OUT["concat(x0, x1, x2, x3)"]
    X0 --> OUT
    L1 --> OUT
    L2 --> OUT

Inside a dense block each layer's input is the concatenation of every earlier output.

The key quantity is the growth rate k: every layer adds only k new channels (for example 32), so a layer's input has c0 + k(l - 1) channels. Because layers can reuse earlier features, each can be thin. DenseNet's building blocks:

  • Dense block: several layers of BN, ReLU, then a 3x3 convolution producing k channels. A 1x1 bottleneck convolution (4k channels) comes before it in the DenseNet-B variant, to save computation.
  • Transition layer: between blocks, a 1x1 convolution that shrinks the channels (by a compression factor of 0.5 in the DenseNet-C variant) followed by 2x2 average pooling, which halves the size. Concatenation only works when sizes match, so pooling can happen only between blocks.
  • DenseNet-BC uses both bottleneck and compression.

The popular variants differ in how many layers each of the four blocks has:

Variant Layers per dense block Total layers
DenseNet-121 6, 12, 24, 16 121
DenseNet-169 6, 12, 32, 32 169
DenseNet-201 6, 12, 48, 32 201
DenseNet-264 6, 12, 64, 48 264
Python Channel bookkeeping and parameters of DenseNet-121

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

DenseNet-121 ends with 1,024 channels and about 8 million parameters (the code gives a close estimate; the official model also counts batch-norm statistics and has small differences), compared with about 25 million for ResNet-50 and 138 million for VGG-16. That efficiency is the headline result.

Advantages and limits of DenseNet

Advantages

  • Alleviates vanishing gradients: every layer has a short path to the loss, so gradients flow to early layers directly.
  • Feature reuse: later layers can use early, low-level features without relearning them, so layers can be narrow, and the parameter count is low.
  • Regularising effect: with fewer parameters it overfits less on small datasets.

Limitations

  • High memory use: all the concatenated feature maps must be kept for the backward pass, so memory grows quickly with depth and image size, even though the parameter count is small. Frameworks offer a "memory-efficient" mode that recomputes some maps.
  • Slower than parameter counts suggest: many small layers and constant concatenation are less GPU friendly than a few big ones.
  • More complex to implement and tune than plain stacks of layers.

Typical uses are medical imaging and other small-data problems, image classification, segmentation backbones (for example the "Tiramisu" network) and feature extractors.

What to take from these designs

Idea Seen in Why it lasts
Shrink space, grow channels LeNet and everything after Keeps computation balanced
Small 3x3 filters, stacked VGG Same receptive field, fewer weights, more non-linearity
Replace big dense heads by global average pooling NiN, ResNet, DenseNet Removes most parameters
Shortcuts (add or concatenate) ResNet, DenseNet Makes depth trainable
Bottlenecks (1x1 convolutions) GoogLeNet, ResNet, DenseNet Cheap channel mixing

In practice you rarely train these from scratch. You load a model pretrained on ImageNet and adapt it, as shown in Pretrained Models and Transfer Learning.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What pattern in LeNet-5 became standard in later CNNs?

    Convolution and pooling reduce height and width while more filters add channels, so deeper layers hold smaller but richer maps.
  2. 2. What are the main design rules of VGG?

    Uniform stacks of 3x3 convolutions with pooling between blocks give VGG its regular structure.
  3. 3. Where does most of VGG-16's parameter count sit?

    The flattened 7x7x512 vector feeding a 4096 unit layer holds over 100 million of the roughly 138 million weights.
  4. 4. How does a DenseNet layer combine earlier feature maps?

    DenseNet concatenates, so the number of input channels grows by the growth rate with every layer.
  5. 5. What does the transition layer between DenseNet blocks do?

    Concatenation needs equal sizes within a block, so downsampling happens in the transitions between blocks.

Summary

  • LeNet-5 (1998) established conv-pool-conv-pool-dense with about 60,000 parameters; AlexNet (2012) scaled it with ReLU, dropout and GPUs.
  • VGG-16 and VGG-19 use only 3x3 convolutions in five blocks with pooling and doubled channels; they have 138 and 144 million parameters, mostly in the dense head.
  • DenseNet concatenates every earlier feature map inside a dense block; the growth rate controls how many channels each layer adds, and transition layers compress and downsample between blocks.
  • DenseNet is parameter-efficient and eases gradient flow, but uses a lot of memory.
  • Counting shapes and parameters with a few lines of Python is the fastest way to understand and compare architectures.

Next: Training a Small Network on Images.