Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 46 ยท Detection, Segmentation and Video

Deep Segmentation: FCN, SegNet, PSPNet, FPN and Panoptic

Understand how segmentation networks upsample and gather context: transposed convolution, SegNet unpooling, PSPNet pyramid pooling, FPN, Mask R-CNN, panoptic merging and GrabCut.

Segmentation Metrics and the U-Net Idea showed that a segmentation network has an encoder that shrinks the image and a decoder that grows it back. This lesson opens the box. Three questions decide how a segmentation model behaves:

  1. How does the decoder grow a coarse map back to full size? (transposed convolution, bilinear upsampling, SegNet's pooling indices)
  2. How does a pixel learn about the whole scene? A pixel's label depends on context: road next to cars, sky above. (dilated convolutions, PSPNet's pyramid pooling, FPN's multi-scale merging)
  3. How do we count individual objects, not just label pixels? (instance and panoptic segmentation, Mask R-CNN)

You will code the key building block of each idea in NumPy, then meet the older, classical alternative, GrabCut, which solves interactive foreground extraction with a graph cut instead of a network.

Three kinds of segmentation

Type Output Two touching cars become
Semantic One class label per pixel One "car" region
Instance A separate mask for every object of a "thing" class Two masks, "car 1" and "car 2"
Panoptic Every pixel gets a class and an instance id; "stuff" (sky, road) has no instances Two cars plus sky plus road, no overlaps

Semantic segmentation suits scene understanding (road, sidewalk, vegetation). Instance segmentation suits counting and tracking. Panoptic segmentation, introduced in 2018, merges the two views so that every pixel is explained exactly once. Applications range from autonomous driving and medical image analysis (organs, tumours) to satellite and aerial mapping, agriculture (crops versus weeds), photo editing (portrait mode, background replacement) and augmented reality.

Fully convolutional networks

A classifier ends in dense layers, which fix the input size and throw away position. The Fully Convolutional Network (FCN, 2014) replaces the dense layers by 1x1 convolutions, so the network accepts any size and outputs a map of class scores, one per location, at a lower resolution (for example 1/32 of the input). The output must then be upsampled to the input size, and how this is done matters for edge quality.

Upsampling: nearest, bilinear, transposed convolution

  • Nearest neighbour repeats each value: fast, blocky.
  • Bilinear blends neighbours: smooth, but fixed, not learned.
  • Transposed convolution ("deconvolution") is a learned upsampling: every input value multiplies a small kernel and the shifted results overlap and add. It can make checkerboard artefacts, so many networks use bilinear upsampling followed by an ordinary convolution instead.
Python Upsampling methods compared

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Nearest gives sharp blocks, bilinear gives a smooth ramp between 1, 3, 5 and 9, and the transposed convolution produces a 5 by 5 result whose overlapping stamps add up (look at the middle rows and columns, where neighbouring stamps overlap and add). In a real network the kernel weights are learned, so the layer can invent sharper boundaries than interpolation could.

SegNet: remember where the maxima were

SegNet (2015) is an encoder-decoder built for road scenes. Instead of copying whole feature maps to the decoder (as U-Net does), it saves only the indices of the maximum in each pooling window. The decoder unpools by placing each value back at its remembered position and leaving zeros elsewhere, then convolutions fill in the gaps. That keeps boundary positions while using much less memory than skip connections.

Python Max pooling with indices, and unpooling

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The unpooled map is sparse but exact where it matters: each maximum is back at the spot it came from. The following convolution layers in the decoder turn this sparse map into a dense prediction.

Context: pyramid pooling (PSPNet)

A pixel in a small patch of fur may look the same whether it belongs to a cat or a rug. To decide, the network needs global context. Two common ways to widen a pixel's view are dilated (atrous) convolutions, which space the kernel taps apart, and the Pyramid Scene Parsing Network (PSPNet, 2017) with its pyramid pooling module:

  1. Take the backbone's feature map (say 60 x 60 x 512).
  2. Pool it at several grid sizes in parallel: 1x1 (the whole image), 2x2, 3x3 and 6x6 bins.
  3. Reduce the channels of each pooled map with a 1x1 convolution.
  4. Upsample every pooled map back to 60 x 60.
  5. Concatenate them with the original features. Now every pixel carries both fine local detail and coarse context at four scales.
flowchart LR
    F["Feature map"] --> P1["pool 1x1"]
    F --> P2["pool 2x2"]
    F --> P3["pool 3x3"]
    F --> P4["pool 6x6"]
    P1 --> U1["conv + upsample"]
    P2 --> U2["conv + upsample"]
    P3 --> U3["conv + upsample"]
    P4 --> U4["conv + upsample"]
    F --> C["concatenate"]
    U1 --> C
    U2 --> C
    U3 --> C
    U4 --> C
    C --> H["conv head: class per pixel"]

Each branch sees the scene at a different scale; concatenation gives every pixel all of them.

Python A pyramid pooling module in NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The coarsest branch (1x1) is constant across the image: it tells every pixel what the whole scene looks like on average. The 6x6 branch varies from region to region. PSPNet's lesson is that long-range context fixes confusions such as "boat" in a scene full of water versus "car" on a road.

Feature Pyramid Network (FPN)

Objects appear at wildly different sizes, and deep layers have strong meaning but low resolution, while shallow layers have fine detail but weak meaning. The Feature Pyramid Network (2017) builds a pyramid that is strong at every level:

  1. Bottom-up pathway: the usual backbone, producing maps at 1/4, 1/8, 1/16 and 1/32 resolution.
  2. Top-down pathway: start from the deepest map and repeatedly upsample by 2.
  3. Lateral connections: add the 1x1-reduced backbone map of the same size. The sum is smoothed with a 3x3 convolution.

Each pyramid level now has high-level meaning at the right resolution. FPN is "plug and play": it sits on top of ResNet or another backbone and is used inside Faster R-CNN, Mask R-CNN, RetinaNet and many segmentation models (Panoptic FPN).

Python Top-down merging of an FPN

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All four outputs share a channel count, so one shared head can be applied to every level, a small head for large objects on P5 and the same head for small objects on P2.

From semantic to instance: Mask R-CNN

Mask R-CNN (2017) extends the two-stage detector Faster R-CNN (see Object Detection Basics). A Region Proposal Network suggests candidate boxes; for each box a head predicts the class and refines the box; and a third head predicts a small binary mask (for example 28 by 28) inside the box. A technique called RoIAlign samples features with bilinear interpolation, so masks align to pixels. Because each object gets its own mask, two touching cars are separate instances.

A panoptic result merges instance masks and a semantic map. The merge rules: sort the instance masks by confidence, paste them in that order without overwriting earlier ones, then fill every unclaimed pixel with the semantic label of its "stuff" class.

Python Merging instances and semantic labels into a panoptic map

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The semantic map shows one merged car region; the panoptic map has two segments (201, 202), the touching column going to the higher-scoring car. Panoptic quality metrics (PQ) score the result by matching predicted and true segments with IoU above 0.5.

Classical alternative: GrabCut and graph cuts

Not every task needs training. GrabCut (2004) extracts a foreground object from a rectangle drawn around it. It models the colours of foreground and background with mixtures of Gaussians, and labels pixels by solving a minimum cut on a graph: every pixel is a node, linked to a "foreground" and a "background" terminal with a cost for each label (how unlikely the pixel's colour is under each model), and linked to its neighbours with a cost for giving them different labels (cut where the image has strong edges, not through smooth regions). The minimum cut is the cheapest way to separate the terminals, so it is the best compromise between colour evidence and smoothness. The two steps (re-fit colour models, re-cut) repeat until stable.

The next demo solves the graph-cut step on a one-row image with SciPy's max-flow routine.

Python Foreground/background by minimum cut

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With no smoothness term (or a small one, 20) every pixel simply takes the label of its nearest colour model, so the dark pixel at position 4 (value 14) between bright pixels becomes a hole in the object. Only when the penalty for switching labels is large (200) does the cut decide it is cheaper to label that noisy pixel foreground, and the object becomes one connected run. The cut cost grows with the penalty because it now includes the price of every label change. OpenCV's cv2.grabCut runs this idea in 2D with colour mixtures, and the related watershed algorithm (see Clustering and Region Segmentation) floods from markers instead. Deep models have replaced both for most recognition tasks, but GrabCut and watershed remain useful when no training data exists.

Practical challenges

  • Boundaries. Low-resolution outputs blur object edges, which is why decoders, skip connections and refinement steps exist.
  • Small and thin objects (poles, wires) vanish when pooling shrinks the map.
  • Class imbalance: the background dominates, so use Dice or focal losses (see Segmentation Metrics).
  • Annotation cost: a pixel-accurate mask takes minutes per image, so datasets are small and often use semi-automatic labelling.
  • Complex backgrounds and domain shift, and compute cost at high resolution.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What distinguishes instance segmentation from semantic segmentation?

    Semantic segmentation labels pixels by class; instance segmentation also distinguishes each object, so two cars get two masks.
  2. 2. What does SegNet's decoder use to unpool?

    Remembering where each maximum came from lets the decoder put values back at the right place cheaply.
  3. 3. What is the purpose of PSPNet's pyramid pooling module?

    Pooling at several grid sizes, upsampling and concatenating gives each pixel scene-level context at several scales.
  4. 4. What do the lateral connections do in a Feature Pyramid Network?

    Top-down maps carry strong semantics but are blurry; lateral connections add precise spatial detail from the backbone.
  5. 5. In GrabCut, what does the minimum cut on the pixel graph represent?

    Cutting terminal links costs colour mismatch, and cutting neighbour links costs label changes; the minimum cut balances both.

Summary

  • Semantic segmentation labels pixels by class, instance segmentation separates objects, and panoptic segmentation combines both into one map.
  • FCNs output class maps at low resolution; decoders upsample with nearest, bilinear or learned transposed convolutions, and SegNet unpools using stored max-pooling indices.
  • PSPNet adds context by pooling at several grid sizes, upsampling and concatenating; FPN merges coarse semantic maps with fine backbone maps through top-down and lateral connections.
  • Mask R-CNN adds a mask head to a two-stage detector; panoptic merging resolves overlaps by score.
  • GrabCut formulates foreground extraction as a minimum graph cut, which needs no training data.
  • Thin objects, boundaries, class imbalance and annotation cost are the recurring practical problems.

Next: Motion and Tracking.