Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 46 ยท Detection, Segmentation and Video

Semantic Segmentation and Its Metrics

Understand semantic segmentation: masks as label arrays, pixel accuracy, IoU and Dice from a confusion matrix, class imbalance, the U-Net idea and mask cleanup.

A bounding box says "the dog is somewhere in this rectangle". Semantic segmentation is far more precise: it labels every pixel of the image with a class such as road, car, sky or tumor. The result is a mask, an array the same height and width as the image, where each entry is a class number. Self-driving systems use it to find drivable surface, doctors use it to outline organs, and satellite tools use it to measure forests and buildings.

Because the output is an image, the metrics are different from classification and detection. In this lesson you will treat masks as ordinary NumPy label arrays, compute pixel accuracy, IoU and Dice from a confusion matrix, see why accuracy is dangerously misleading when the object is small, trace the shapes inside a U-Net, and clean up a noisy predicted mask. Box IoU from Object Detection Basics carries over directly, just counted over pixels instead of rectangle areas.

Masks are label arrays

In a semantic mask, a pixel stores a class index. Class 0 is usually background. A mask for three classes in a 14 by 28 image is an int array of that shape holding 0, 1 or 2. No new data structure is needed: a mask is just an image whose "color" is a label, and all the array tools you know (slicing, boolean comparison, counting with bincount) apply.

The demo builds a ground-truth mask with a disc (class 1) and a rectangle (class 2) and a "prediction" that is slightly off: the disc is shifted by one pixel diagonally and the rectangle is a little too small. It prints both masks as text and the pixel accuracy.

Python Two masks and pixel accuracy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The prediction looks reasonable and gets 90.6% of the pixels right (37 wrong out of 392). Pixel accuracy is the share of pixels whose predicted label equals the true label, and it is the simplest segmentation metric. The last line shows why it is not enough: 71.2% of the image is background, so even a mask that is partly wrong keeps a high score. We need metrics that look at each class separately.

The confusion matrix, IoU and Dice

Count, for every pair of classes, how many pixels have true class i and predicted class j. That table is the confusion matrix. With label arrays it takes one line: encode each pixel pair as k * truth + pred and count with np.bincount. For each class, three numbers then follow from the matrix:

  • TP: true class and predicted class are both this class (the diagonal).
  • FP: predicted this class but it was another (column sum minus the diagonal).
  • FN: truly this class but predicted another (row sum minus the diagonal).
flowchart LR
    T["true mask"] --> C["confusion matrix<br/>k x k counts"]
    P["predicted mask"] --> C
    C --> D["per class:<br/>TP, FP, FN"]
    D --> E["IoU = TP / (TP + FP + FN)"]
    D --> F["Dice = 2TP / (2TP + FP + FN)"]

Every segmentation metric in this lesson is a formula over the same three counts.

IoU (Jaccard index) divides the overlap by the union; Dice (also called the F1 score) divides twice the overlap by the sum of both areas. They always rank predictions in the same order and are linked by Dice = 2 * IoU / (1 + IoU). Dice is always at least as large as IoU, which is why a paper that reports Dice can look more impressive than one that reports IoU for the same masks. The mean IoU (mIoU) averages IoU over classes and is the standard single number for multi-class segmentation.

Python Confusion matrix, per-class IoU and Dice

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The matrix shows exactly where the errors are: 11 true disc pixels were called background, 11 background pixels were called disc, and 15 true box pixels were lost to the background. The per-class view is much less flattering than the 90.6% accuracy: the shifted disc has IoU 0.633 and the too-small box 0.766. The background scores 0.879 simply because it is large and forgiving. The mIoU of 0.759 is a more honest summary than accuracy. Notice that the disc and the box never confuse with each other, so the zeros off the background row and column are a good sign.

Note: Some benchmarks ignore the background class when averaging, or ignore pixels labeled "void" at object borders. Check the evaluation protocol before comparing two numbers: mIoU with and without background are not the same metric.

Class imbalance

In many real problems the object of interest is tiny: a lesion in a scan, a crack in a road, a ship in a sea image. If the object covers 2% of the pixels, a model that predicts background everywhere is 98% accurate and completely useless. The experiment below makes this concrete with a 14 by 14 object in a 100 by 100 image and three different predictions.

Python Why accuracy fails on small objects

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The "all background" prediction wins on accuracy (0.980) and has IoU and Dice of exactly 0, since it finds no object pixels. A box shifted by only 3 pixels still has accuracy 0.985 but a modest IoU of 0.446: small objects punish small misalignments hard. The bloated box (0.340 IoU) loses on accuracy too, but accuracy barely separates it from the shifted one (0.962 against 0.985) while IoU shows a clear gap. The practical rules are:

  • Report per-class IoU or Dice, not just accuracy, whenever classes are unbalanced.
  • Train with a loss that cares about overlap or re-weights rare classes: Dice loss, focal loss, or a weighted cross-entropy where rare classes get a larger weight.
  • Sample training crops so that images with the rare class are not drowned by empty ones.

The U-Net idea

How does a network produce a mask the size of the input? A plain classification CNN shrinks the image step by step until only a vector is left, which throws away where things are. Segmentation networks add a decoder that grows the feature maps back to full size. The classic design is the U-Net, first developed for biomedical images, and it is still the basis of countless medical and industrial models.

flowchart LR
    A["input<br/>64x64"] --> B["encoder<br/>conv + pool<br/>shrinks, more channels"]
    B --> C["bottleneck<br/>8x8, many channels"]
    C --> D["decoder<br/>upsample + conv<br/>grows back"]
    D --> E["output<br/>one score per class<br/>per pixel"]
    B -. "skip connections<br/>copy fine detail" .-> D

The U shape: the encoder understands what is in the image, the decoder puts it back at full resolution, and skip connections carry sharp edges across.

The encoder is a CNN like the ones in Pooling and CNN Architectures: convolutions and pooling shrink the map while the channel count grows, so deeper features know more about content and less about exact position. The decoder reverses the shrinking with upsampling (interpolation or a transposed convolution). The skip connections are the key trick: the high-resolution feature map from each encoder level is concatenated onto the matching decoder level, giving the decoder the fine details (exact edges) that were lost while pooling. A final 1 by 1 convolution turns the last feature map into one score per class per pixel, and argmax over classes gives the label mask.

Python Tracing shapes through a small U-Net

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

After the bottleneck (128 channels at 8 by 8), the decoder doubles the size three times until it is back at 64 by 64, concatenating the saved maps of 64, 32 and 16 channels along the way. Input size must be divisible by two as many times as there are pooling steps, otherwise the shapes of the skip and upsampled maps will not match; this is a classic source of errors. A framework model is read-only here (it would need PyTorch), but its structure is exactly this trace. Alternatives you will meet include DeepLab (dilated convolutions to keep resolution), SegFormer (transformers), and promptable models that return a mask for a clicked point.

Tip: Semantic segmentation gives one label per pixel, so two touching cars become one "car" region. Instance segmentation separates individual objects, and panoptic segmentation combines both. If you need to count objects, you either need an instance model or a classical step such as connected components on the mask.

Cleaning up a predicted mask

Raw network outputs are rarely perfect: isolated wrong pixels ("salt-and-pepper" speckles), small holes inside objects, ragged borders. Cheap post-processing with the tools you already know can fix much of this, but you should measure each step, because not every step helps. The example below builds a ground-truth mask (a disc plus a bar), flips 6% of the pixels at random, punches a hole in the disc, and then applies three steps: fill holes, remove components under 20 pixels, and a morphological opening.

Python Mask post-processing, one step at a time

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The raw mask scores IoU 0.840. Filling holes lifts it to 0.920 (the hole and a few interior gaps vanish). Of the 17 connected components that remain, 15 are single stray pixels; keeping only components of at least 20 pixels leaves the two real regions (148 and 120 pixels) and raises IoU to 0.971. The third step is an instructive failure: the opening lowers IoU to 0.945, because a 3 by 3 opening also shaves the corners of the rectangular bar and the thin tips of the disc. The lesson is to validate each cleanup step on labeled data instead of applying a standard recipe blindly. Morphological Operations explains why opening behaves this way, and Connected Components and Labeling covers the labeling used in step two.

Other common post-processing steps are keeping only the largest component (when you know there is exactly one object), smoothing probabilities before taking the argmax, and a conditional random field or other edge-aware refinement for sharper boundaries.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A segmentation mask for 5 classes of a 256x256 image has what shape and content?

    A semantic mask has the same height and width as the image and stores one class index per pixel. (A network may output (5, 256, 256) scores first, then take argmax over the class axis.)
  2. 2. The object covers 1% of an image and the model predicts background everywhere. Which statement is true?

    Nearly every pixel is background, so accuracy is high. But there are no true positives for the object class, so its IoU and Dice are both 0.
  3. 3. A class has TP = 30, FP = 10 and FN = 20. What are its IoU and Dice?

    IoU = 30 / (30 + 10 + 20) = 0.50. Dice = 60 / (60 + 10 + 20) = 0.667. This also matches Dice = 2 * IoU / (1 + IoU).
  4. 4. What is the purpose of skip connections in a U-Net?

    Pooling discards precise positions. Concatenating the encoder's high-resolution feature maps onto the decoder restores sharp edges.
  5. 5. In the post-processing example, why did the morphological opening make the IoU worse?

    Opening erodes and then dilates, which removes small features. When the true shapes have sharp corners, those are removed as well, so every cleanup step should be measured.

Summary

  • Semantic segmentation assigns a class to every pixel; the mask is an integer array the size of the image.
  • Pixel accuracy is simple but misleading when classes are unbalanced. Compute per-class IoU and Dice from a confusion matrix (TP, FP, FN) and report mIoU.
  • IoU = TP / (TP + FP + FN); Dice = 2TP / (2TP + FP + FN); Dice = 2 * IoU / (1 + IoU).
  • For rare classes use overlap-based or weighted losses and smart sampling.
  • A U-Net has an encoder that shrinks, a decoder that grows, and skip connections that carry fine detail; input sizes must divide cleanly by the pooling factor.
  • Post-processing (hole filling, removing small blobs, opening) can raise IoU, but test every step, since some steps hurt.

Next up: add time to the picture with Motion, Optical Flow and Tracking.