Skip to content
Tutorials Center
My learning

Topics

Lesson 69 of 73 ยท Projects

Project: Image Captioning

Build an image captioning model in NumPy: a CNN encoder feeds an LSTM decoder, trained with teacher forcing, then evaluate captions with exact match and BLEU.

Describe a photograph in a sentence: "a dog catching a frisbee in a park". People do it effortlessly, but it combines two very different skills: seeing (what is in the picture and where) and writing (turning that into fluent language). Image captioning teaches a network to do both, and it is the classic first project that joins computer vision to natural language processing. The same encoder-decoder pattern underlies modern systems that answer questions about images or describe them in detail.

The classic design, from the "Show and Tell" paper, has two parts. A CNN encoder turns the image into a feature vector. An LSTM decoder then writes the caption one word at a time, starting from that vector, exactly like the sequence-to-sequence decoder in Seq2Seq, Teacher Forcing and Beam Search, but with an image instead of a sentence on the input side. In this project you build both parts in NumPy and train them on a tiny captioning problem you can solve end to end.

Two networks, one pipeline

Encoder A CNN turns the image into a feature vector.
Decoder An LSTM writes the caption word by word.
Link The image features set the decoder's initial state.
1 / 2

The data: pictures with sentences

Real caption data sets pair photographs with five human-written sentences each: Flickr8k has about 8,000 images, and MS COCO Captions has more than 100,000. We cannot load photographs here, so the project uses generated 12 by 12 images that contain one shape (a disc, square, cross or ring) that is dim or bright, placed in one of four places (left, right, top, bottom). Each image comes with a caption built from a small vocabulary of 16 tokens, for example a bright ring on the top. The captions have to be right on three facts at once, which is a miniature of what real captioning requires.

The pipeline has two stages:

  1. Encoder. We take the CNN from Training a CNN From Scratch and pretrain it on a related task, classifying the shape. Then we freeze it and keep its 288-number feature vector for each image. Real captioning systems do exactly this with a CNN pretrained on ImageNet (see Transfer Learning and Fine-Tuning).
  2. Decoder. An LSTM whose initial hidden state is a learned projection of the image features. It is fed the start token <s> and the previous true word at each step (teacher forcing), and trained with cross-entropy to predict the next word, finishing with an end token </s>. At test time it writes greedily, feeding back its own predictions.

The backward pass is the LSTM BPTT you have written before, with one extra step at the end: the gradient arriving at the decoder's initial state flows into the projection of the image features, which is how the decoder learns what the image features mean.

Python A CNN encoder and an LSTM decoder that writes captions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The encoder, trained only to name the shape, reaches 71.0% shape accuracy on new images (it is small and trained briefly). The caption loss falls from 2.77 to 0.02 over 250 steps. On 200 unseen images the decoder gets the whole caption exactly right 76.0% of the time. The word-by-word breakdown is the useful part: brightness is right 99.0% of the time, place 93.5% and shape only 81.5%. The decoder is not the weak link, since it writes fluent sentences every time. The bottleneck is what the frozen encoder passes along: shape is the feature that is hardest to read from this small CNN, and the captions' errors are visible in the examples (a dim square on the bottom written as a dim disc on the bottom). A caption can be no better than the visual features that feed it. The remedies are the ones from the CNN projects: a stronger or better-pretrained encoder, more data, and fine-tuning the encoder together with the decoder.

Judging captions

Exact match is harsh: the sentence "a shiny round shape at the left" is a perfectly good caption of a bright disc on the left, yet it would count as wrong. Real data sets therefore have several reference captions per image, and the standard automatic metrics compare word overlap with them. BLEU (originally for machine translation) counts how many of the candidate's n-grams (single words, word pairs, triples and so on) also appear in a reference, clips repeated matches, and multiplies by a brevity penalty that punishes candidates much shorter than the references. Related metrics include METEOR (handles synonyms and stems), ROUGE and CIDEr (designed for captioning: it weights n-grams that are distinctive of an image) and SPICE (compares the objects and relations in a scene graph).

Python BLEU, exact match and why you need both

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Reading the table: the perfect copy and the synonym rewording both score 1.00 because each is one of the references. A caption with one wrong word (square for disc) keeps BLEU-1 at 0.83 and BLEU-2 at 0.71 but its BLEU-4 is 0.00, because no four-word run matches, so a single factual error is treated very differently depending on the metric. The too-short caption, a bright disc, is entirely correct but gets 0.37, held down by the brevity penalty. Fluent nonsense, which repeats the correct phrase three times, scores 0.50 on BLEU-1 and 0.42 on BLEU-4, well above the wrong-shape caption, because repeated correct n-grams still match. And the unrelated sentence gets 0.17 on BLEU-1. The lessons: BLEU tracks overlap and not truth, it can reward the wrong things, and it is only meaningful with several references and averaged over many captions. Always read samples, and for important systems add human evaluation or check specific facts (object, count, place) the way our word-by-word accuracy did.

Decoding, attention and modern captioners

  • Greedy versus beam search. We decoded greedily. Beam search (keep the best k partial captions) usually gives slightly better captions, with the length-normalization caveat from the seq2seq lesson.
  • Attention. The "Show, Attend and Tell" model keeps the spatial grid of CNN features (not one flattened vector) and uses attention so that, while writing each word, the decoder looks at the relevant part of the image. It improves captions and gives a map of where the model looked.
  • Transformers. Modern captioners use a vision transformer or a CLIP-style image encoder and a transformer decoder (see The Transformer Block). Large vision-language models caption, answer questions and follow instructions with the same encoder-decoder idea at far larger scale.

The real-photo recipe

On photographs the classic model looks like this in PyTorch. This code is read-only.

Python
# Read-only: a "Show and Tell" captioner in PyTorch (needs torchvision and a captions data set such as Flickr8k)
import torch, torch.nn as nn
from torchvision import models

class Captioner(nn.Module):
    def __init__(self, vocab_size, embed=256, hidden=512):
        super().__init__()
        cnn = models.resnet50(weights=models.ResNet50_Weights.DEFAULT)
        self.encoder = nn.Sequential(*list(cnn.children())[:-1])           # drop the classifier: output (B, 2048, 1, 1)
        for p in self.encoder.parameters():
            p.requires_grad = False                                         # frozen at first, as in the NumPy version
        self.to_state = nn.Linear(2048, hidden)                             # image features -> initial LSTM state
        self.embed = nn.Embedding(vocab_size, embed)
        self.lstm = nn.LSTM(embed, hidden, batch_first=True)
        self.out = nn.Linear(hidden, vocab_size)

    def forward(self, images, words_in):                                    # words_in: <start> w1 w2 ... (teacher forcing)
        feats = self.encoder(images).flatten(1)
        h0 = torch.tanh(self.to_state(feats)).unsqueeze(0)
        out, _ = self.lstm(self.embed(words_in), (h0, torch.zeros_like(h0)))
        return self.out(out)                                                # logits for the next word at every position

# Loss: cross-entropy of logits[:, :-1] against words[:, 1:], ignoring padding. At test time decode with beam search.

Preparing the text is as important as the model: lowercase and tokenize the captions, build a vocabulary of the words that occur at least five times (the rest become <unk>), add <start>, <end> and <pad> tokens, and pad each batch. See Hugging Face Transformers Pipelines for subword tokenizers, which avoid the unknown-word problem altogether.

Pitfalls and responsibility

  1. Hallucination. Captioners confidently describe objects that are not in the image, since the decoder is a strong language model that prefers likely sentences. Check object-level accuracy, not only fluency.
  2. Bias. Models learn the biases of their captions: stereotyped descriptions of people, genders and activities. Evaluate on varied images and be careful with captions of people.
  3. Generic captions. Systems drift toward safe, frequent phrases ("a man standing in a room"). Diversity metrics and CIDEr reward specificity.
  4. Accessibility is a real use. Alt-text for the visually impaired is a worthwhile application, but errors matter, so say when a caption is machine-generated.
  5. Leakage. Several captions belong to each image; split by image, never by caption.

Ideas to extend the project

  • Add beam search to the decoder and compare exact-match accuracy with greedy decoding.
  • Fine-tune the encoder jointly with the decoder, and see whether the shape word improves.
  • Add a count to the images ("two discs on the left") and watch which words become hardest.
  • Give each image three different phrasings as references and evaluate with BLEU.
  • Replace the single flattened feature vector with the spatial feature grid and add attention.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. How does the image information reach the LSTM decoder in this project?

    The CNN features are passed through a learned layer to initialize the LSTM's memory, and the decoder then generates words conditioned on that state.
  2. 2. The decoder wrote fluent captions, but the shape word was right only 81.5% of the time. What does that suggest?

    The language side was fine. The error comes from weak visual features, which a stronger or fine-tuned encoder would improve.
  3. 3. Why is exact match a poor metric for captions on its own?

    There are many valid wordings. Metrics such as BLEU compare overlap with several references, though they have weaknesses of their own.
  4. 4. What does the brevity penalty in BLEU do?

    Without it, a one-word caption that happens to match would get perfect precision. The penalty prevents that.
  5. 5. What is a hallucination in image captioning?

    Strong language priors can make the decoder write plausible but false details. Evaluate object-level correctness, not only fluency.

Summary

  • Image captioning pairs a CNN encoder with an LSTM decoder; the image features initialize the decoder, which writes words one at a time.
  • Training uses teacher forcing and cross-entropy; at test time the decoder feeds on its own words (greedy or beam search).
  • On the toy task the model got 76.0% of captions exactly right: brightness 99.0%, place 93.5%, shape 81.5%. The weakest word pointed to the weakest part of the encoder.
  • Exact match is too strict and BLEU only measures overlap: a wrong shape cost little in BLEU-1 and drastically in BLEU-4, while repeated phrases and very short captions behaved oddly. Use several references and read samples.
  • Real captioners use pretrained CNN or transformer encoders, attention over image regions, beam search, and need care about hallucination and bias.

Next project: Machine Translation with a Transformer.