Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 46 ยท Neural Networks for Vision

Transfer Learning and Modern Vision Models

Reuse pretrained vision models: feature extraction versus fine-tuning, how vision transformers cut images into patches, and what foundation models add.

Training a modern vision network from scratch takes millions of labeled images and days of GPU time. Almost nobody does that for their own project. Instead, you start from a model that someone has already trained on a huge dataset and transfer what it learned. Early layers of such a network detect edges, textures and simple shapes that are useful in almost every image task, so you only need to adapt the last part to your classes.

In this lesson you will see why transfer learning works, run a small honest stand-in for it in your browser, compare feature extraction with fine-tuning by counting trainable parameters, and look at how vision transformers and foundation models changed the field. The browser cannot run PyTorch, so the framework code is read-only and every runnable example is a labeled stand-in built from NumPy and scikit-learn. The earlier lessons on pooling and architectures and training a small network explain the building blocks.

Why transfer works

A CNN trained on a large photo collection learns a hierarchy: the first layers respond to edges and color blobs, the middle layers to textures and parts, and the last layers to whole objects specific to the training classes. The first two levels are generic. The last one is specific. Transfer learning keeps the generic part and replaces or retrains the specific part.

flowchart LR
    A["image"] --> B["early layers<br/>edges, colors"]
    B --> C["middle layers<br/>textures, parts"]
    C --> D["late layers<br/>task-specific objects"]
    D --> E["head<br/>your classes"]

Generic features sit early in the network; the head is the part you almost always replace.

You have two main strategies:

  • Feature extraction (frozen backbone): keep every pretrained weight fixed, run your images through the network once, and train only a small new head (often logistic regression or one linear layer) on the resulting feature vectors. Fast, cheap, and hard to overfit.
  • Fine-tuning: start from the pretrained weights but let some or all layers keep learning on your data, with a small learning rate. More accurate when you have enough data, but slower and easier to overfit.

A stand-in you can run

We cannot download a photo-trained network here, so the demo builds the same situation on handwritten digits. First a "backbone" is trained to read the digit label on 900 images. Then, for a different task (is the digit odd or even?) on images the backbone has never seen, we train only a logistic-regression head on a few labeled examples. Three feature sets compete: the raw 64 pixels, the 32 hidden units of an untrained (random) network of the same size, and the 32 hidden units of the pretrained backbone. The random twin is a useful control: it shows whether the gain comes from learning or merely from having more or different features.

Python Frozen pretrained features versus raw pixels

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each cell is the average test accuracy over five random draws of the labeled examples. With only 10 labeled images the pretrained features reach 78.4%, against 74.6% for raw pixels and 73.3% for random features. At 30 labels the lead is 80.1% against 77.2%. With 100 labels it is gone (87.7% against 88.6% for raw pixels), which is the typical pattern: transfer helps most when your labeled data is scarce, and with enough labels plain pixels catch up on an easy dataset like this one. The random control stays at or below the raw-pixel score in every row, so the early gain came from what the backbone learned, not from the extra features alone. Notice also that the backbone was never trained for odd versus even; the digit-reading task taught it features that happen to be useful for it.

Note: Real transfer from photos is far more dramatic than this miniature, because a backbone trained on over a million varied images has seen edges, fur, wheels and leaves. The principle is identical: train on a big, general task, then reuse the features on a small, specific one.

Feature extraction versus fine-tuning

How much of the network you train changes the cost and the risk. The helper below counts trainable parameters for a ResNet-18-style classifier under four strategies. The backbone has 11,176,512 parameters; the original head maps 512 features to 1,000 ImageNet classes, and we replace it with a 10-class head.

Python How many parameters do you train?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Feature extraction trains 5,130 weights, about 0.05% of the model, so it needs very little data and no GPU memory for gradients through the backbone. Fine-tuning only the last block trains 42.3% of the weights, because the deepest layers are also the widest. Full fine-tuning trains everything.

flowchart TD
    A["how much labeled data<br/>do you have?"] -->|"very little"| B["feature extraction<br/>frozen backbone + linear head"]
    A -->|"some, similar domain"| C["fine-tune last blocks<br/>small learning rate"]
    A -->|"a lot, or very different domain"| D["fine-tune everything<br/>or train from scratch"]

A rule of thumb. The more your images differ from the pretraining data (medical scans, satellite pictures), the deeper you need to fine-tune.

In PyTorch the whole recipe is short. The snippet below is read-only because the browser has no PyTorch; it freezes a torchvision ResNet-18, swaps the head for a 5-class one, and trains it:

Python
import torch
from torch import nn
from torchvision import models

weights = models.ResNet18_Weights.DEFAULT
model = models.resnet18(weights=weights)           # pretrained on ImageNet
preprocess = weights.transforms()                  # resize, crop, normalize like training

# --- feature extraction: freeze everything, new head ---
for p in model.parameters():
    p.requires_grad = False
model.fc = nn.Linear(model.fc.in_features, 5)      # new layer is trainable by default
optimizer = torch.optim.Adam(model.fc.parameters(), lr=1e-3)

# --- fine-tuning: unfreeze the last block, use a smaller learning rate there ---
for p in model.layer4.parameters():
    p.requires_grad = True
optimizer = torch.optim.Adam([
    {"params": model.layer4.parameters(), "lr": 1e-5},
    {"params": model.fc.parameters(),     "lr": 1e-3},
])

Two details trip people up. First, you must feed images through the same preprocessing the backbone was trained with (size, color order, mean and standard deviation); weights.transforms() does this for you. Second, pretrained layers already hold good weights, so use a learning rate 10 to 100 times smaller for them than for the new head, or the first noisy updates will destroy what they learned.

Warning: Check the license and the training data of a pretrained model before shipping a product with it. Some weights are research-only, and a model inherits the biases and blind spots of the data it was trained on (see Pitfalls, Ethics and Deployment).

Vision transformers

For a decade CNNs ruled vision. The Vision Transformer (ViT, 2020) showed that a model borrowed from language processing can match them when it has enough training data. The idea: cut the image into a grid of square patches (for example 16 by 16 pixels), flatten each patch, project it to a vector called a token, add a position embedding so the model knows where each patch came from, and let a stack of transformer layers process the sequence. Each layer uses self-attention: every token compares itself with every other token and mixes in information from the ones that matter.

flowchart LR
    A["image"] --> B["cut into<br/>patches"]
    B --> C["flatten and project<br/>to tokens"]
    C --> D["add position<br/>embeddings"]
    D --> E["transformer layers<br/>self-attention"]
    E --> F["class token<br/>to head"]

A ViT turns the image into a sequence of patch tokens and lets attention relate them all to each other.

The demo cuts an 8 by 8 array of consecutive numbers into four 4 by 4 patches (so you can see which pixels land in which patch), embeds them, and computes one self-attention step by hand. The weights are random: a real model learns them. Then it prints how the number of tokens, and the size of the attention matrix, grows with image size.

Python Patches, tokens and one attention step

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Row i of the attention matrix says how much token i listens to each of the four patches; each row sums to 1. The table at the bottom explains a key trade-off: a 224 by 224 image with 16 pixel patches gives 196 tokens and a 38,416-entry attention matrix per head, but a 512 by 512 image gives 1,024 tokens and over a million entries. Attention cost grows with the square of the number of tokens, so high-resolution images are expensive, and many designs add windows or hierarchies (such as Swin) to tame it.

How do ViTs compare with CNNs? A CNN has a built-in assumption that nearby pixels matter and that a feature is the same everywhere (see The Convolutional Layer). That prior helps when data is limited. A ViT has far fewer assumptions, so it needs more data (or a strong pretrained start) but scales very well and sees the whole image from the first layer. In practice you pick whichever pretrained backbone works best for your task and budget; modern CNNs and ViTs are close in accuracy.

Swin Transformer: attention in windows

The attention table above showed the problem: global attention cost grows with the square of the number of tokens. The Swin Transformer (2021, "shifted windows") solves it with two ideas:

  1. Window-based self-attention. Tokens only attend to the other tokens inside a small window (for example 7 by 7 = 49 tokens). The cost grows linearly with image size, because the number of windows grows linearly.
  2. Shifted windows. Alternate layers move the window grid by half a window, so tokens near a window border in one layer are in the middle of a window in the next, and information flows across the whole image.

It also builds a hierarchy: patch merging layers halve the resolution and double the channels between stages, like a CNN, so Swin produces feature maps at several scales. That makes it a drop-in backbone for detection and segmentation (it plugs into FPN, Mask R-CNN and UPerNet), which plain ViT is not.

Python Global versus windowed attention cost

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Doubling the image side multiplies the global pairs by 16 but the windowed pairs only by 4, so the advantage keeps growing. The costs are a more complicated implementation (window partitioning, shift masks) and less direct long-range attention in each single layer: global context arrives only after several layers of shifting and merging.

Foundation models

A foundation model is a very large model pretrained on a vast, broad dataset, usually without hand-made class labels, and then reused for many tasks. In vision you meet a few kinds:

  • Self-supervised backbones (DINOv2 is a well-known example) learn features by matching different views of the same image. Their frozen features work as the "backbone" in the feature extraction recipe above, often with just a linear head.
  • Image-text models (CLIP and its relatives) train an image encoder and a text encoder so that a picture and its caption land near each other in one shared space. This enables zero-shot classification: embed the picture, embed the sentences "a photo of a cat" and "a photo of a dog", and pick the closer one, with no training on your classes.
  • Promptable segmentation models (the Segment Anything family) produce object masks from a click or a box.

The zero-shot idea is just cosine similarity between vectors. A read-only sketch with the Hugging Face library:

Python
from transformers import CLIPModel, CLIPProcessor
from PIL import Image

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

image = Image.open("photo.jpg")
labels = ["a photo of a cat", "a photo of a dog", "a photo of a bicycle"]
inputs = processor(text=labels, images=image, return_tensors="pt", padding=True)
probs = model(**inputs).logits_per_image.softmax(dim=1)   # one probability per label

Foundation models are convenient, but they are large, may be slow to run, and carry the biases of web-scale data. Before using one, measure it on a small labeled sample of your images; the evaluation lesson shows how.

Tip: A sensible workflow: (1) extract features with a strong frozen backbone and train a linear head; (2) check the validation score; (3) only if it is not enough, fine-tune the last blocks; (4) only if you have lots of data, fine-tune everything. Each step costs more, so stop as soon as the result is good enough.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In feature extraction, which weights are trained?

    The backbone is frozen and acts as a fixed feature extractor. Only the small new head is trained on your labeled data.
  2. 2. In the digit experiment, why was a random (untrained) network of the same size included as a baseline?

    If the random twin matches the pretrained features, the benefit would be due to feature expansion. It did not, so what the backbone learned mattered.
  3. 3. Why is a much smaller learning rate used for pretrained layers during fine-tuning?

    The pretrained weights are already good. Gentle updates adapt them; large ones can wipe out the knowledge you wanted to reuse.
  4. 4. A ViT splits a 224x224 image into 16x16 patches. How many tokens does it get (ignoring any extra class token)?

    224 / 16 = 14 patches per side, and 14 x 14 = 196 tokens.
  5. 5. What makes zero-shot classification with an image-text model possible?

    Because pictures and captions were trained to land close together, you can classify by picking the class description whose embedding is most similar to the image embedding.

Summary

  • Pretrained backbones hold generic features (edges, textures, parts) that transfer to new tasks; transfer helps most when labeled data is scarce.
  • Feature extraction freezes the backbone and trains a small head; fine-tuning also updates some or all backbone layers with a small learning rate.
  • Always reuse the preprocessing (size, normalization) the backbone was trained with, and compare against a simple baseline.
  • Vision transformers cut images into patches, embed them as tokens with position information, and use self-attention; the cost grows with the square of the token count.
  • Foundation models (self-supervised backbones, image-text models, promptable segmentation) enable few-shot and zero-shot use but need honest evaluation on your own data.
  • Escalate gradually: frozen features, then last-block fine-tuning, then full fine-tuning.

Next up: from one label per image to boxes around many objects in Object Detection Basics.