Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 46 ยท Detection, Segmentation and Video

Object Detection Basics

Learn how detectors find objects: bounding boxes, IoU and non-maximum suppression written from scratch, sliding windows, anchors, and box precision and recall.

Classification answers one question about a whole image: "what is in it?" Object detection answers two harder ones for every object: "what is it?" and "where is it?" The answer to "where" is a bounding box, a rectangle around the object, and a detector outputs a list of them, each with a class label and a confidence score. Detection powers face finders, traffic cameras, quality inspection on factory lines and the "tap to identify" feature in a photo app.

You cannot understand a detector, or judge whether it works, without a few small tools: a way to measure how well two boxes agree (IoU), a way to remove duplicate boxes (non-maximum suppression), a way to propose candidate boxes (sliding windows and anchors), and metrics that count hits and misses for boxes. In this lesson you will write all of them in NumPy. Large detectors such as YOLO and Faster R-CNN are described in read-only form, because they need a GPU framework that does not run in the browser.

Boxes and coordinates

A box needs four numbers. The most common formats are:

  • Corner format (x1, y1, x2, y2): top-left and bottom-right corners. We use this one.
  • Center format (cx, cy, w, h): the center point plus width and height. Many models predict this form.
  • Corner plus size (x, y, w, h): used by several datasets.

As with images, the vertical axis points down: y grows toward the bottom of the picture, and (0, 0) is the top-left pixel corner. Mixing formats up is the single most common bug in detection code, so always write the format in a comment or a variable name.

flowchart LR
    A["image"] --> B["detector"]
    B --> C["box 1: person<br/>score 0.97"]
    B --> D["box 2: dog<br/>score 0.88"]
    B --> E["box 3: dog<br/>score 0.41"]

A detector returns a variable-length list of (box, class, score) triples, not a fixed-size vector.

Intersection over Union

How do you tell whether a predicted box matches the true box? Comparing the four numbers directly is awkward: being off by 5 pixels is a disaster for a tiny object and nothing for a huge one. The standard measure is Intersection over Union (IoU), also called the Jaccard index: the area where the two boxes overlap, divided by the area covered by either of them.

TEXT
IoU = area(A and B) / area(A or B) = intersection / (area A + area B - intersection)

IoU is 1 when the boxes are identical, 0 when they do not overlap, and it is independent of the image scale. The code below computes the intersection rectangle by taking the larger of the two left and top edges and the smaller of the two right and bottom edges; if that rectangle has no positive width or height, the boxes are disjoint.

Python IoU from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A box shifted by a quarter of its width already drops to 0.600, and a box that covers exactly half of the truth scores 0.500. A common convention is that a prediction is correct when its IoU with a true box is at least 0.5. Stricter benchmarks average over thresholds from 0.5 up to 0.95, which rewards precise localization. A box that is too big (0.444 above) is penalized just like one that is too small, because the union grows.

Note: IoU is a single number, so it hides how a box is wrong. Two boxes with IoU 0.6 can be shifted, too large or too small. When debugging a detector, look at the boxes drawn on the images, not only at the averages.

Non-maximum suppression

A detector usually produces many overlapping boxes around the same object, each with a slightly different score. Reporting all of them would count one dog as five dogs. Non-maximum suppression (NMS) keeps the best box and throws away the ones that overlap it too much:

  1. Sort boxes by score, highest first.
  2. Keep the top box. Remove every remaining box whose IoU with it exceeds a threshold.
  3. Repeat with the highest-scoring remaining box until none are left.
flowchart TD
    A["sorted boxes<br/>by score"] --> B["take the best box<br/>and keep it"]
    B --> C["drop all boxes with<br/>IoU above threshold"]
    C --> D{"boxes left?"}
    D -->|yes| B
    D -->|no| E["kept boxes"]

Greedy NMS. The threshold decides how much overlap counts as "the same object".

The version below computes the whole pairwise IoU matrix once with broadcasting, then runs the greedy loop. The six boxes form three clusters: three near (10, 10), two near (80, 20) and one alone.

Python Non-maximum suppression

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With a threshold of 0.5 the result is [3, 0, 5]: one box per cluster, each the highest-scoring in its group. With a threshold of 0.9 almost nothing counts as a duplicate, so all six boxes survive. With 0.1, boxes need to barely overlap to be removed; here it gives the same answer as 0.5 only because the clusters are far apart. In crowded scenes (people shoulder to shoulder) a low threshold suppresses real neighbors, which is a known weakness of greedy NMS; variants such as Soft-NMS lower scores instead of deleting boxes.

Sliding windows

Before deep learning, detection was done with a simple idea: train a classifier for "object" versus "not object" on a fixed-size patch, then slide that patch over every position of the image and score each one. A sliding window detector has three costs: the number of positions, the number of window sizes (objects come in different scales), and the cost of each classification.

The demo below places a bright 12 by 12 square on a noisy dark image and scores each window by its mean brightness, a stand-in for a classifier.

Python Sliding a window over an image

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three lessons hide in the output. First, stride 1 finds the exact position (row 14, column 30, mean 0.900), but stride 4 and 8 miss the object by a few pixels and score only about 0.66: a coarse stride trades accuracy for speed. Second, 117 windows around the single square score above 0.45, which is precisely why detectors need NMS. Third, searching three window sizes at stride 2 already means 810 classifications on this tiny image; on a real photo with a deep network per window it is far too slow. That cost pushed researchers to share computation between windows, which leads to anchors and modern detectors.

Anchors

Instead of sliding a window and classifying each one from scratch, modern detectors run the network once over the image and make predictions at every cell of a feature map grid. At each grid position they place a set of reference boxes called anchors, with several scales and aspect ratios. For each anchor, the network predicts "is there an object here?" and a small correction to the anchor's shape.

During training, each true box is matched to anchors by IoU: anchors with a high IoU (say 0.5 or more) become positives and learn to predict that object plus the correction, anchors with a low IoU (below 0.4, say) become negatives, and anchors in between are ignored. The correction is expressed as offsets relative to the anchor so that it does not depend on object size: shifts divided by the anchor width and height, and log-ratios for the size.

Python Anchors and regression targets

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The 576 anchors (an 8 by 8 grid, three sizes, three shapes) include 12 positives for the single true box and 550 clear negatives. That imbalance (a handful of positives against hundreds of background anchors) is typical, and it is why detectors need tricks such as sampling negatives or a special loss that down-weights easy background. The best anchor is a 32 by 32 square with IoU 0.672; the targets say "move 6% of the width right, 12.5% of the height down, keep the width, and shrink the height by a factor of exp(-0.288) = 0.75". At inference the network predicts those offsets and the code applies them in reverse.

How real detectors are organized

Detectors fall into two broad families (read-only here, since they need a deep learning framework):

  • Two-stage detectors, such as the R-CNN family and Faster R-CNN. A first stage proposes a few hundred regions that probably contain something (the Region Proposal Network scores anchors). A second stage crops each proposal from the shared feature map and classifies it and refines the box. They are accurate and handle small objects well, but slower.
  • One-stage detectors, such as YOLO ("you only look once") and SSD. They skip the proposal step and predict class scores and box offsets directly for every grid cell or anchor in a single pass. They are fast enough for video on modest hardware. Newer designs (anchor-free detectors, transformer-based detectors such as DETR) remove anchors or even NMS, but the ideas you have just coded still describe the evaluation.

A typical usage of a pretrained detector looks like this (read-only; PyTorch and torchvision are not available in the browser):

Python
import torch
from torchvision import models
from torchvision.ops import box_iou, nms

weights = models.detection.FasterRCNN_ResNet50_FPN_Weights.DEFAULT
model = models.detection.fasterrcnn_resnet50_fpn(weights=weights).eval()

image = torch.rand(3, 480, 640)                    # a stand-in for a real photo
with torch.no_grad():
    out = model([image])[0]                        # dict: boxes, labels, scores

keep = out["scores"] > 0.5                         # drop low-confidence boxes
print(out["boxes"][keep])                          # corner format (x1, y1, x2, y2)
# torchvision.ops.box_iou and nms are the library versions of the code above

With the Ultralytics library a YOLO model is about three lines (YOLO("yolov8n.pt")(path)), and its result object exposes the same boxes, classes and scores. The post-processing, thresholds and metrics are the part you tune.

Inside the Region Proposal Network and SSD

Two ideas from those families are worth seeing in numbers.

Region Proposal Network (RPN). At every position of a shared feature map the RPN places a set of anchors (typically 3 scales times 3 aspect ratios = 9). For each anchor a small head predicts two things: an objectness score (is there any object here, ignoring the class?) and four box offsets that adjust the anchor to fit. Anchors with IoU above 0.7 with a ground-truth box are labelled positive, those below 0.3 negative, the rest ignored. NMS then trims the thousands of scored anchors to a few hundred proposals that the second stage classifies.

Single Shot MultiBox Detector (SSD). SSD predicts class scores and box offsets directly from several feature maps of different resolution: high-resolution maps look for small objects, the low-resolution ones for large objects. Each cell of each map has a few default boxes (anchors). Training minimises a weighted sum of a localization loss (smooth L1 between predicted and true offsets, only for matched positive boxes) and a confidence loss (cross-entropy over classes, with a background class). Because almost every default box is background, SSD uses hard negative mining: keep all positives and only the negatives with the highest loss, at a ratio of about 3 negatives per positive.

Python SSD default boxes and hard negative mining

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

SSD300 always has 8,732 default boxes, almost all of them background, which is why the unbalanced loss needs mining. Only the matched positives contribute to the localization loss. SSD's trade-off is speed (one forward pass) against a weaker result on very small objects, which have little signal on the coarse maps; the Feature Pyramid Network in Deep Segmentation Networks is the standard remedy.

Precision and recall for boxes

A detection is a true positive (TP) if it matches an unmatched true box with IoU above a threshold, a false positive (FP) if it matches nothing (a phantom, a duplicate or a poorly placed box), and each true box left without a match is a false negative (FN). Then:

TEXT
precision = TP / (TP + FP)      "of the boxes I drew, how many were right?"
recall    = TP / (TP + FN)      "of the real objects, how many did I find?"

Matching must be greedy and one-to-one: go through predictions from highest score down, and let each true box be claimed only once, so duplicates count as false positives. Sweeping the score cutoff from high to low traces a precision-recall curve: a strict cutoff gives high precision and low recall, a lenient cutoff finds more objects but also adds false alarms. The area under that curve is average precision (AP); the mean of AP over classes is mAP, the headline number on most detection benchmarks.

Python Matching, precision, recall and AP

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At IoU 0.5 the detector finds all three objects (recall 1.00) but also draws two bad boxes (a phantom at (60, 60) and a duplicate), so precision is 0.60. At IoU 0.9 only the exact duplicate is accurate enough, so recall collapses to 0.33. The curve shows the trade-off: at cutoff 0.90 precision is still 1.00 with recall 0.67, then the 0.80 phantom pulls precision to 0.67, and the last correct box lifts recall to 1.00 at precision 0.60. The AP of 0.867 is the area under the precision envelope (the best precision achievable at each recall or higher). Always say which IoU threshold an AP number refers to; "AP at 0.5" and "AP averaged over 0.5 to 0.95" are very different scores.

Warning: Duplicates are false positives. If you forget the one-to-one rule and let two predictions match the same true box, precision looks better than it is. Also compute metrics per class; a high average can hide a class the detector never finds.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Box A is (0, 0, 10, 10) and box B is (5, 0, 15, 10). What is their IoU?

    The intersection is 5 x 10 = 50. Each box has area 100, so the union is 100 + 100 - 50 = 150, and 50 / 150 = 1/3.
  2. 2. What problem does non-maximum suppression solve?

    Detectors fire on many nearby windows or anchors. NMS keeps the highest-scoring box and removes boxes that overlap it above a threshold.
  3. 3. A detector draws 10 boxes, 6 of which are correct, and there are 12 real objects. What are precision and recall?

    Precision = TP / predictions = 6 / 10 = 0.6. Recall = TP / real objects = 6 / 12 = 0.5.
  4. 4. Why do anchor-based detectors regress offsets relative to an anchor instead of absolute box coordinates?

    Dividing shifts by the anchor's width and height, and using log size ratios, makes the targets scale-free, so one head can handle objects of different sizes.
  5. 5. Two predicted boxes both overlap the same true box with IoU 0.8. How should a one-to-one evaluation count them?

    A true box can be claimed only once. The best-scoring prediction takes it; the second one is a duplicate, which counts as a false positive.

Summary

  • A detector outputs a variable number of (box, class, score) triples. Keep track of the box format and the direction of the y axis.
  • IoU is intersection area divided by union area; 0.5 is the common "correct" threshold, and stricter thresholds reward tight boxes.
  • Non-maximum suppression keeps the best box in each cluster of overlapping boxes. Its threshold trades duplicates against merged neighbors.
  • Sliding windows score many positions and sizes, which is too slow with a deep network. Anchors share computation: the network predicts a class score and a relative correction for every anchor of a grid.
  • Two-stage detectors (Faster R-CNN) are accurate; one-stage detectors (YOLO, SSD) are fast. Newer designs drop anchors or NMS but keep the same evaluation.
  • Precision, recall, the precision-recall curve and AP are computed with greedy one-to-one matching at a stated IoU threshold; mAP averages AP over classes.

Next up: the same idea at pixel level, where every pixel gets a label, in Semantic Segmentation and Its Metrics.