Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 73 ยท Convolutional Networks

Transfer Learning and Fine-Tuning

Learn transfer learning: reuse pretrained features, freeze layers, fine-tune with small learning rates, and see when it helps, with a NumPy experiment.

Training a good image network from nothing takes a huge labeled data set and days of GPU time. Most projects have neither. Transfer learning is the way out: start from a network that someone already trained on a large data set, keep what it learned, and adapt it to your own small problem. A photo classifier pretrained on a million images has already discovered edges, textures and shapes, and those are useful for nearly every other visual task.

This lesson explains the standard recipe (reuse the body, replace the head, optionally fine-tune), when it works and when it does not, and then lets you measure the effect yourself. We cannot download a real pretrained model in the browser, so the experiments use a small NumPy network pretrained on one digit task and transferred to another. The framework code for real models is shown read-only.

The transfer learning recipe

Pretrained body Layers trained on a big data set. They know general features.
Freeze Keep the body fixed so it is not damaged.
New head Train a fresh final layer for your own classes.
Fine-tune Optionally unfreeze and adjust the body gently.
1 / 2

What a pretrained network gives you

Look at a trained CNN layer by layer. The first convolutions learn simple, general patterns: edges, color blobs, corners. Middle layers combine them into textures and parts. The last layers are the most specialized: they respond to the particular classes of the original data set.

So a network trained on one task contains a feature extractor (the body) followed by a classifier (the head). The body's output, a compact vector of features, is often useful for entirely different labels. Transfer learning reuses the body and replaces only the head.

The original task is called the source task and yours the target task. The more similar they are, the more transfers: photos to photos works wonderfully, photos to X-rays works partly, photos to audio spectrograms works less well.

Two ways to adapt a pretrained network

Feature extraction (frozen body). Remove the old head, attach a new one with the number of outputs you need, freeze the body (its weights receive no updates) and train only the head. This is cheap, fast, and safe on tiny data sets because the number of learnable parameters is small. The body can even be run once to precompute the features.

Fine-tuning. After the head has learned something sensible, unfreeze some or all body layers and keep training the whole network with a small learning rate. The body gently specializes to your data. The small rate matters: the new head starts out random, and large, noisy gradients from it would otherwise wreck the carefully learned body weights in the first steps. A typical recipe is therefore two phases:

flowchart LR
    A["Pretrained network"] --> B["Replace the head"]
    B --> C["Phase 1: freeze body<br/>train the head"]
    C --> D["Phase 2: unfreeze<br/>tiny learning rate on the body"]
    D --> E["Your model"]

Train the head first, then adjust everything softly.

Common refinements are discriminative learning rates (the lower a layer sits, the smaller its rate) and unfreezing only the last few blocks. If your data set is tiny, stop after phase 1.

An experiment: pretrain, freeze, compare

Here is the whole workflow with a miniature stand-in, using the digits data from scikit-learn (see Images as Tensors).

  • Source task: a network with one hidden layer of 64 ReLU units learns to recognise all ten digits from 1,200 images. This is our "pretrained model".
  • Target task: decide whether a digit is even or odd, using the remaining images, but with only a handful of labeled examples (3 per class).

We compare four strategies, each trained with the Adam optimizer (see Momentum, RMSProp and Adam) and averaged over three random draws of the small training set:

  1. scratch: random weights, train everything.
  2. random frozen: a random, never-trained body with a trained head (a baseline for "how much do the learned features matter?").
  3. frozen pretrained: pretrained body, trained head.
  4. fine-tuned: pretrained body with a body learning rate ten times smaller than the head's.

The optimizer takes one learning rate per parameter, and a rate of zero is how freezing is done. The last column verifies that a frozen body really did not change.

Python Scratch versus frozen versus fine-tuned

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With only 3 labeled examples per class, training from scratch reaches about 59% on a two-class problem (chance is 50%). Freezing the pretrained body and training just a new head reaches about 76%. The random frozen body scores about 57%, so the gain comes from the learned features, not from merely having a big fixed layer. Fine-tuning at a tenth of the head's rate lands close to the frozen result (about 74%). The last column confirms that the frozen runs left the body untouched, while the other two changed it.

How much data do you need?

Transfer learning matters most when labeled data is scarce. With abundant data, a network trained from scratch eventually learns good features itself, and the advantage shrinks. The next experiment varies the number of labeled examples per class and plots the test accuracy for each strategy.

Python The advantage shrinks as data grows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The shape is typical. With 2 examples per class, scratch training is barely above chance (about 54%) while the pretrained body already gives about 70%. As the data grows the gap closes, and at 30 examples per class the from-scratch network (about 90%) has caught up with the frozen body (about 85%). Fine-tuning keeps the head start and still adapts, ending near the scratch result (about 89%). Each point averages only two random draws, so small differences between the lines are noise; the large gap at the left is the real effect.

How gently should you fine-tune?

The body's learning rate is the key setting of fine-tuning. Zero is plain feature extraction; a very large value erases what was learned. The next experiment multiplies the body's learning rate by different factors (relative to the head's) on a 5-per-class target set.

Python Body learning-rate multiplier

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A body rate of zero (pure feature extraction) and the small rates 0.01 and 0.1 give about 83%. When the body learns as fast as the head (x1) accuracy drops to about 78%, and at ten times the head's rate to about 77%: with so few examples, large body updates overwrite the pretrained features with noise. That is why fine-tuning uses small body rates, and why the head is often trained alone first.

Negative transfer. If source and target are too different, the borrowed features can be useless or even harmful, and training from scratch (or from a closer source) wins. Check a pretrained model against a simple baseline on your own validation data; never assume.

The same recipe in a framework

With real pretrained models, freezing means turning off gradients, and replacing the head means swapping the last layer. These snippets are read-only: PyTorch, torchvision and Keras are not available in the browser, and the pretrained weights would have to be downloaded.

Python
# PyTorch + torchvision (read-only)
import torch
import torch.nn as nn
from torchvision import models

model = models.resnet18(weights=models.ResNet18_Weights.DEFAULT)   # pretrained on ImageNet

for param in model.parameters():          # phase 1: freeze the whole body
    param.requires_grad = False

model.fc = nn.Linear(model.fc.in_features, 5)    # new head for 5 classes (trainable)

optimizer = torch.optim.Adam(model.fc.parameters(), lr=1e-3)
# ... train for a few epochs ...

for param in model.layer4.parameters():   # phase 2: unfreeze the last block
    param.requires_grad = True

optimizer = torch.optim.Adam([            # discriminative learning rates
    {"params": model.layer4.parameters(), "lr": 1e-5},
    {"params": model.fc.parameters(),     "lr": 1e-4},
])
Python
# Keras (read-only)
import keras

base = keras.applications.MobileNetV2(weights="imagenet", include_top=False,
                                      input_shape=(160, 160, 3))
base.trainable = False                    # freeze the body

model = keras.Sequential([
    base,
    keras.layers.GlobalAveragePooling2D(),
    keras.layers.Dense(5, activation="softmax"),     # new head
])
model.compile(optimizer=keras.optimizers.Adam(1e-3),
              loss="sparse_categorical_crossentropy", metrics=["accuracy"])
# model.fit(...), then base.trainable = True, recompile with a tiny learning rate (1e-5), fit again

A few practical details apply to every framework:

  • Use the same preprocessing as the pretrained model: image size and the same normalization (mean and standard deviation) it was trained with.
  • Keep normalization layers in mind. When the body is frozen, put it in evaluation mode so batch norm keeps using its stored running statistics.
  • Recompile or re-create the optimizer after changing what is trainable.
  • Pretraining is not only for images: the same recipe powers language models, which are pretrained on text and fine-tuned for a task (see How Large Language Models Work).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In feature extraction with a frozen body, which parameters are updated during training?

    The body is frozen, so it receives no updates. Only the newly added head, which starts random, is learned.
  2. 2. Why use a small learning rate when fine-tuning the pretrained layers?

    The pretrained weights are already good. Gentle updates adapt them without erasing what they learned.
  3. 3. When does transfer learning give the biggest advantage?

    With little data, reusing good features saves the model from having to learn them. With plenty of data, training from scratch catches up.
  4. 4. What is negative transfer?

    If the source task is too different, its features may not apply, and a model trained from scratch can do better.

Summary

  • A pretrained network is a feature extractor (body) plus a task-specific classifier (head); transfer learning keeps the body and replaces the head.
  • Feature extraction freezes the body and trains only the new head. Fine-tuning then unfreezes some layers and trains them with a much smaller learning rate.
  • Early layers are general and transfer well; late layers are specific, so they are the first to unfreeze or replace.
  • Transfer helps most with small target data sets and related tasks; the advantage fades as labeled data grows, and unrelated tasks can cause negative transfer.
  • In practice, match the pretrained model's preprocessing, keep frozen normalization layers in evaluation mode, and compare against a from-scratch baseline.

Next: Object Detection.