Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 73 ยท Convolutional Networks

Image Segmentation

Learn semantic and instance segmentation, build a U-Net style encoder-decoder with skip connections in NumPy, and see why Dice loss beats accuracy.

Object detection draws a rectangle around each object. A rectangle is a rough answer: it also contains road, sky and background, and it cannot follow the outline of a curved object. Many tasks need the exact outline. A medical tool must measure the area of a tumor, a self-driving car needs to know which pixels are drivable road, and a photo app cuts a person out of the background.

Image segmentation assigns a label to every pixel. Instead of a handful of boxes, the output is a map the same size as the image. This lesson covers the kinds of segmentation, the encoder-decoder design called U-Net that dominates the field, the two operations a decoder needs (upsampling and transposed convolution), and the Dice loss, which fixes a problem accuracy hides.

Three flavors of segmentation

Semantic Every pixel gets a class. All cars share one label.
Instance Each object gets its own mask. Car 1 and car 2 differ.
Panoptic Both at once. Things are split, background is labeled.

Semantic answers "what", instance answers "which one".

1 / 2

What the output looks like

For semantic segmentation into K classes, the network outputs an array of shape (K, H, W): a score for each class at each pixel. A softmax over the class axis turns the scores into probabilities, and an argmax picks one class per pixel. The training label is an (H, W) integer map of the true classes, and the usual loss is cross-entropy averaged over all pixels. For a single object type such as "tumor or not", K = 1 with a sigmoid, and the label is a 0/1 mask.

Instance segmentation must separate objects of the same class. Mask R-CNN extends Faster R-CNN (see the detection lesson): after a region is proposed and its box refined, a small branch predicts a mask for that box. Each detection thus carries its own mask, so two overlapping people stay distinct. Panoptic segmentation combines both: countable "things" (people, cars) get instance masks, and uncountable "stuff" (sky, road, grass) gets semantic labels.

The encoder-decoder idea

A classifier shrinks the image step by step until only a vector of class scores remains, as in CNN Architectures. That throws away the "where" on purpose. Segmentation needs the "where" back. The natural design is two halves:

  • The encoder is a normal CNN: convolutions and pooling cut the height and width (say from 64 to 8) while the channel count grows. Deep features know what is in the image and see a wide context, but they are blurry.
  • The decoder reverses the shrinking: it upsamples back to 64 by 64 and ends with one score per class at every pixel.

The first fully convolutional networks (FCN) did exactly this, but their outputs were coarse, because the decoder had to invent fine detail from a tiny 8 by 8 map. U-Net (2015, originally for biomedical images) fixed it with skip connections: at every resolution, the decoder receives a copy of the encoder's feature map from the same level and concatenates it along the channel axis. Early encoder features are sharp and high resolution (edges, boundaries), and the decoder combines them with the deep, meaning-rich features coming up from below. The drawn shape is a "U".

flowchart TB
    A["Input 1x32x32"] --> E1["Encoder level 1<br/>8 ch, 32x32"]
    E1 --> E2["Encoder level 2<br/>16 ch, 16x16"]
    E2 --> BN["Bottleneck<br/>32 ch, 8x8"]
    BN --> U2["Upsample to 16x16"]
    E2 -. "skip: concatenate" .-> C2["Decoder level 2<br/>16+32 in, 16 out"]
    U2 --> C2
    C2 --> U1["Upsample to 32x32"]
    E1 -. "skip: concatenate" .-> C1["Decoder level 1<br/>8+16 in, 8 out"]
    U1 --> C1
    C1 --> O["1x1 conv: K class scores<br/>K x 32 x 32"]

A two-level U-Net. Dotted arrows are skip connections that concatenate encoder maps into the decoder.

Growing the map back: upsampling and transposed convolution

Pooling and strided convolutions shrink a map. The decoder needs the opposite, and there are two common tools.

Fixed upsampling copies values: nearest-neighbor repeats each pixel into a block, and bilinear interpolates smoothly. It has no weights; a normal convolution after it learns to clean up the result.

Transposed convolution (sometimes wrongly called "deconvolution") has learnable weights. Where a stride-2 convolution reads a window and writes one number, a stride-2 transposed convolution reads one number and writes a whole kernel-sized patch into the output, adding where patches overlap. The output is about twice as large. The kernel decides what each input pixel "paints". Because uneven overlap of patches can leave a checkerboard pattern, many modern decoders prefer upsampling followed by a 3 by 3 convolution.

Python Two ways to double a feature map

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With a 2 by 2 kernel and stride 2 the painted patches tile the output without overlap, so the transposed convolution with an all-ones kernel reproduces nearest-neighbor upsampling exactly. With a 3 by 3 kernel and stride 2 the patches overlap, the output is 5 by 5 ((2 - 1) * 2 + 3), and the middle row and column receive contributions from several input pixels (the center value is 1 + 2 + 3 + 4 = 10). A learned kernel would shape these contributions instead of simply summing them.

Tracing a U-Net

The next program wires up a complete two-level U-Net forward pass with random weights. It is untrained, so its output is meaningless, but it shows every shape and where the skip connections plug in. It reuses the convolution and pooling ideas from Convolution, Pooling and Feature Maps: a 3 by 3 "same" convolution keeps the size, pooling halves it, upsampling doubles it.

Python A tiny U-Net forward pass: shapes and skips

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The output follows the whole U: 32 by 32 down to 8 by 8 and back, with channels growing 8, 16, 32 and shrinking again. Because the weights are random, the predicted mask is meaningless (here every pixel lands in the same class); training would shape the weights so the mask matches the labels.

Notice how the channel counts at the decoder levels (16 + 32 = 48 and 8 + 16 = 24 on entry) reflect the concatenation. Without the skips, each decoder level would receive only the upsampled map; with them, it also sees fine-resolution detail that the pooling threw away. The final 1 by 1 convolution simply mixes the channels at each pixel into K scores.

Measuring segmentation: why accuracy is not enough

Pixel accuracy, the fraction of pixels labeled correctly, sounds natural and is misleading. A small tumor may cover 2% of a scan. A model that paints everything as "background" gets 98% accuracy and finds nothing. Two overlap measures work better. Let P be the predicted foreground pixels and T the true ones:

TEXT
IoU  = |P and T| / |P or T|
Dice = 2 * |P and T| / (|P| + |T|)

IoU is the Jaccard index from the detection lesson, now computed on pixels. Dice (also the F1 score on pixels) is similar and always at least as large as IoU; the two are linked by Dice = 2 * IoU / (1 + IoU). Both ignore the huge number of correctly labeled background pixels, so they punish the all-background model completely.

For training, a differentiable version is used as a loss. The soft Dice loss replaces the hard 0/1 prediction with the network's probabilities p and the intersection with a sum of products:

TEXT
dice_loss = 1 - (2 * sum(p * t) + eps) / (sum(p) + sum(t) + eps)

It optimizes the overlap directly and so handles class imbalance far better than plain cross-entropy. Many segmentation recipes use a sum of both (cross-entropy for stable gradients, Dice for overlap).

Python Pixel accuracy versus IoU and Dice

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The all-background prediction scores 97.8% accuracy yet has IoU and Dice of 0, which is the whole point. A prediction shifted by a single pixel keeps accuracy at 97.5% but shows IoU 0.286 and Dice 0.444, so the overlap measures are the ones that reflect the quality of the outline. The identity Dice = 2 * IoU / (1 + IoU) holds exactly. For the soft Dice loss, probabilities close to the truth give a small loss (0.077), while a vague map that spreads 5% probability over every pixel scores 0.969, close to the worst value of 1, even though most of its pixels are "nearly right".

Using a segmentation model in practice

A real U-Net has the same structure with more levels (four is typical), two convolutions per level, batch normalization, and 64 to 1024 channels. In PyTorch it is a few dozen lines, and pretrained encoders from libraries such as segmentation_models_pytorch or torchvision's deeplabv3 models make training on your own data a fine-tuning job, as in Transfer Learning and Fine-Tuning.

Python
# Read-only: PyTorch is not available in the in-browser runner.
import torch
from torch import nn

def block(c_in, c_out):
    return nn.Sequential(
        nn.Conv2d(c_in, c_out, 3, padding=1), nn.BatchNorm2d(c_out), nn.ReLU(),
        nn.Conv2d(c_out, c_out, 3, padding=1), nn.BatchNorm2d(c_out), nn.ReLU(),
    )

class TinyUNet(nn.Module):
    def __init__(self, in_ch=1, classes=1):
        super().__init__()
        self.enc1, self.enc2 = block(in_ch, 16), block(16, 32)
        self.mid = block(32, 64)
        self.pool = nn.MaxPool2d(2)
        self.up2 = nn.ConvTranspose2d(64, 32, 2, stride=2)       # learnable upsampling
        self.dec2 = block(64, 32)                                # 32 (skip) + 32 (up)
        self.up1 = nn.ConvTranspose2d(32, 16, 2, stride=2)
        self.dec1 = block(32, 16)
        self.head = nn.Conv2d(16, classes, 1)

    def forward(self, x):
        e1 = self.enc1(x)
        e2 = self.enc2(self.pool(e1))
        m = self.mid(self.pool(e2))
        d2 = self.dec2(torch.cat([e2, self.up2(m)], dim=1))
        d1 = self.dec1(torch.cat([e1, self.up1(d2)], dim=1))
        return self.head(d1)                                     # logits, (N, classes, H, W)

Two practical notes. The input height and width must be divisible by the total pooling factor (here 4), or the skip connections will not line up; padding or cropping the image fixes this. And the training loss usually adds Dice to a pixel-wise binary or categorical cross-entropy, as described above.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the difference between semantic and instance segmentation?

    In semantic segmentation two cars share the label car. Instance segmentation gives each car its own mask.
  2. 2. What do the skip connections in a U-Net do?

    Pooling discards fine spatial detail. Concatenating the same-level encoder feature map into the decoder gives it that detail back.
  3. 3. A scan has a tumor covering 2% of the pixels. A model predicts 'no tumor' everywhere. Which is true?

    Almost all pixels are background, so accuracy is high. Dice and IoU look only at the foreground overlap, which is empty here.
  4. 4. How does a stride-2 transposed convolution differ from nearest-neighbor upsampling?

    Nearest-neighbor copies values with no parameters. A transposed convolution writes a learned kernel for each input pixel and sums the overlaps.

Summary

  • Segmentation labels every pixel: semantic (class per pixel), instance (one mask per object) or panoptic (both).
  • The output is a (K, H, W) score map; softmax and argmax over the class axis give the predicted mask.
  • Encoder-decoder networks shrink the image to learn context, then upsample back to full size.
  • U-Net concatenates same-level encoder features into the decoder, restoring the sharp detail that pooling removed.
  • Decoders grow maps with fixed upsampling (nearest, bilinear) or learnable transposed convolutions.
  • Pixel accuracy hides class imbalance; IoU and Dice measure foreground overlap, and soft Dice loss trains on it directly.

Next: Working With Sequences.