Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 45 ยท Computer Vision

Transfer Learning and Fine-Tuning

Reuse pretrained networks in PyTorch: load weights, freeze layers, replace the classifier head, set discriminative learning rates and decide when to fine-tune.

Training a deep vision model from scratch needs a huge dataset and a lot of compute. Transfer learning avoids both. You start from a network that was already trained on a large dataset such as ImageNet (about 1.2 million photos in 1,000 classes), keep what it learned about edges, textures and shapes, and adapt only the final part to your own task. With a few hundred images per class and a laptop GPU, or even a CPU, you can often match what would otherwise take weeks of training.

This lesson covers loading pretrained weights, replacing the head, freezing and unfreezing layers, using different learning rates for different parts, and judging when each strategy fits. PyTorch and torchvision cannot run on this site, and pretrained weights need a download, so the PyTorch code is read-only. The live editors use a faithful stand-in for the main idea: a small network "pretrained" on a bigger task (10-way digit recognition with scikit-learn), whose hidden layer is then frozen and reused for a new task with very few labels. It is a toy, not ImageNet, but it shows the real workflow: pretrain, freeze, extract, train a new head.

Why transfer works

Early layers Edges and colors. Useful for any image.
Middle layers Textures and parts. Often reusable.
Last layers Specific to the old task. Replace them.

General features transfer; task-specific ones do not.

1 / 3

Loading a pretrained model

torchvision.models contains many architectures (ResNet, EfficientNet, ConvNeXt, Vision Transformers and more). Recent versions load weights through a weights argument that carries both the parameters and the matching preprocessing.

Python
import torch
from torch import nn
from torchvision import models

weights = models.ResNet18_Weights.DEFAULT        # the best available ImageNet weights
model = models.resnet18(weights=weights)         # downloads once, then cached on disk
preprocess = weights.transforms()                # resize, crop, normalize as used in training

print(model.fc)                                  # Linear(in_features=512, out_features=1000, bias=True)

The preprocess object matters: a pretrained network expects its input prepared exactly as during its original training (image size, and the ImageNet channel means and standard deviations). Feeding it differently normalized images quietly hurts accuracy. The default ImageNet statistics are mean (0.485, 0.456, 0.406) and std (0.229, 0.224, 0.225); see Image Classification Pipeline for how normalization works.

Replacing the head

The last layer of ResNet-18, fc, maps 512 features to 1,000 ImageNet classes. Your task has a different number of classes, so swap that layer for a fresh one:

Python
num_classes = 5
model.fc = nn.Linear(model.fc.in_features, num_classes)   # new, randomly initialized, trainable

Everything before fc still holds pretrained weights, and the new layer starts from a normal random initialization. The attribute name of the classifier differs between architectures (fc for ResNet, classifier for several others, heads for Vision Transformers), so print the model once to find it.

Freezing layers

To keep pretrained parameters fixed, switch off their gradients. Do this before creating the optimizer, and give the optimizer only the parameters that still train:

Python
for param in model.parameters():
    param.requires_grad = False          # freeze everything

model.fc = nn.Linear(model.fc.in_features, num_classes)   # new layer: requires_grad is True again

trainable = [p for p in model.parameters() if p.requires_grad]
print(sum(p.numel() for p in trainable), "trainable parameters")   # 2565 = 512 * 5 + 5
optimizer = torch.optim.AdamW(trainable, lr=1e-3)

This is feature extraction: the backbone acts as a fixed function that turns an image into a 512-dimensional vector, and only the small head learns. It is fast, uses little memory (no gradients for the backbone) and cannot overfit much, which suits small datasets.

There is one trap with frozen layers: batch normalization. Setting requires_grad = False stops the weights from changing, but in train() mode a BatchNorm layer still updates its running mean and variance from your batches. When freezing a backbone with batch norm, keep those layers in evaluation mode:

Python
def set_backbone_eval(model):
    for module in model.modules():
        if isinstance(module, nn.BatchNorm2d):
            module.eval()

model.train()
set_backbone_eval(model)       # call again after every model.train()

Further, since the backbone never changes in pure feature extraction, you can compute its output once for every image, store the vectors, and train the head on them over many fast epochs. That trick is exactly what the live example below does.

Feature extraction in a stand-in

The editor first "pretrains" a small network on 1,000 digit images with ten classes (the big source task). Its hidden layer becomes the frozen backbone. The new task is different: tell odd digits from even ones, with only a handful of labeled images. A logistic regression head is trained three ways: on raw pixels, on a random frozen layer of the same size, and on the pretrained features. Each setting is repeated over 20 random choices of labeled images and averaged. The exact scores depend on the library versions, but the pretrained features should come out ahead at every size.

Python Frozen features plus a trained head (scikit-learn stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Fine-tuning and discriminative learning rates

When your data is larger or quite different from ImageNet, frozen features may not be enough. Fine-tuning lets the pretrained layers move too, but gently: the backbone already contains good weights, and a large learning rate would destroy them in the first few steps (called catastrophic forgetting). The standard remedy is a discriminative learning rate: small for pretrained layers, larger for the new head. PyTorch optimizers support this with parameter groups:

Python
backbone_params = [p for name, p in model.named_parameters() if not name.startswith("fc.")]
optimizer = torch.optim.AdamW([
    {"params": backbone_params,        "lr": 1e-5},   # pretrained: tiny steps
    {"params": model.fc.parameters(),  "lr": 1e-3},   # new head: normal steps
], weight_decay=1e-4)

A popular two-phase recipe combines both ideas: first train only the head for a few epochs until its output is sensible, then unfreeze the backbone (or its last blocks) and continue with the small backbone rate. Training the head first matters because a randomly initialized head sends large, meaningless gradients into the backbone at the beginning. A schedule such as warmup followed by cosine decay (see Learning Rate Schedules and Gradient Clipping) fits fine-tuning well.

A middle path is partial fine-tuning: freeze the early blocks, which hold generic edges and textures, and unfreeze only the last one or two:

Python
for name, param in model.named_parameters():
    param.requires_grad = name.startswith(("layer4.", "fc."))

For ResNet, layer4 is the last residual stage. Early layers stay frozen, which saves memory and keeps general features intact.

Gradual unfreezing, simulated

The second editor simulates the effect of a learning rate on a pretrained weight. A single "pretrained" parameter already sits near a good solution for a related task; we adapt it to a slightly different target. A huge learning rate throws the weight away from the good region, while a small one adapts it smoothly. This is a toy of the real phenomenon, but it shows why the backbone gets the smaller rate.

Python Why pretrained weights need a small learning rate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When to use which strategy

Two questions decide: how much data do you have, and how similar is it to the pretraining data?

  • Little data, similar domain (a few hundred photos of everyday objects): freeze the backbone and train the head. Fine-tuning many millions of parameters on that little data overfits.
  • More data, similar domain: fine-tune all or most layers with discriminative rates.
  • Different domain (medical scans, satellite images, spectrograms): features are less reusable, so unfreeze more layers; with a lot of data, training from scratch can become competitive. Even then, pretrained initialization often converges faster.
  • Very large data: any of the above works; fine-tuning usually still wins on speed.

Also check the practicalities: the preprocessing must match the pretrained model; the licence of the weights must allow your use; and the model size must fit your hardware and latency budget. Keep a separate validation set to compare strategies, because "more fine-tuning" is not automatically better.

Do not evaluate on images the pretrained model might have seen during its own training and then claim a surprising result. When your test images come from a public dataset, they may overlap with ImageNet-style web images.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. You freeze a ResNet backbone and replace its fc layer. Which parameters should you pass to the optimizer?

    Passing only the trainable parameters (the new head) keeps the optimizer state small and avoids touching frozen weights. Frozen parameters have no gradient, so they would not move, but filtering is cleaner.
  2. 2. Why use a smaller learning rate for pretrained layers than for the new head?

    The backbone is already good, so it needs only small adjustments. The random new head starts far from a solution and can take bigger steps.
  3. 3. What is the main reason to call batch norm layers' eval() when the backbone is frozen?

    requires_grad=False does not stop BatchNorm from updating running mean and variance in train mode. Keeping those layers in eval mode preserves the pretrained statistics.
  4. 4. You have 200 labeled images that look like everyday ImageNet photos. Which approach is the safest first choice?

    Small, similar data favors feature extraction: few trainable parameters, low overfitting risk and fast training.

Summary

  • Transfer learning reuses a network pretrained on a large dataset; only the last layers are specific to the old task.
  • Load weights with weights=..., use the matching weights.transforms() preprocessing, and replace the classifier head (fc, classifier or heads).
  • Freeze layers with requires_grad = False before building the optimizer, and keep frozen batch norm layers in eval mode.
  • Fine-tuning uses parameter groups: a small learning rate for the backbone and a larger one for the new head; training the head first is a common safe recipe.
  • Choose the strategy by dataset size and similarity to the pretraining data, and validate the choice.

Next, move from images to ordered data with RNNs, LSTMs and GRUs. For adapting very large models with few trainable parameters, see Fine-Tuning, PEFT and LoRA.