Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 73 ยท Convolutional Networks

Object Detection

From R-CNN to YOLO: learn bounding boxes, IoU, non-maximum suppression and anchor matching, and build each one yourself in NumPy with runnable code.

A classifier answers one question about a whole picture: "what is in it?" Real scenes are messier. A street photo holds three cars, a bicycle and a dozen pedestrians, and you want to know what each thing is and where it sits. That task is object detection: for every object in the image, output a class label and a bounding box, usually with a confidence score.

Detection looks like a harder version of classification, but most of the new difficulty is bookkeeping rather than new math. How do you describe a box? How do you measure whether a predicted box is "close enough" to the truth? How do you stop the network from reporting the same car twelve times? How do you decide which prediction is responsible for which object during training? This lesson answers each question with a small function you can run. (To run a pretrained detector from a photo-processing script, see Deep Learning in OpenCV.)

What a detector outputs

Box Four numbers that frame the object.
Class Car, person, dog, and so on.
Score How sure the network is.
Many objects Zero, one or dozens per image.

The number of outputs changes from image to image.

1 / 2

Boxes and coordinates

A box needs four numbers. Two conventions are common, and detectors convert between them all the time:

  • Corner format: (x1, y1, x2, y2), the top-left and bottom-right corners.
  • Center format: (cx, cy, w, h), the center point plus width and height.

Image coordinates start at the top-left, with y growing downward. In this lesson, boxes are NumPy arrays with one row per box in corner format, and positions are plain numbers (a box from 10 to 50 is 40 units wide).

Intersection over Union

To judge a predicted box against a true one, divide the area they share by the area they cover together. This ratio is intersection over union (IoU), also called the Jaccard index:

TEXT
IoU(A, B) = area(A and B) / area(A or B) = I / (area(A) + area(B) - I)

It is 0 when the boxes do not touch and 1 when they are identical. The overlap is itself a box: its left edge is the larger of the two left edges, its right edge the smaller of the two right edges, and likewise for top and bottom. If that overlap has no positive width or height, the intersection is zero.

The version below compares every box in one array with every box in another, using broadcasting so there are no Python loops. You will reuse it in the next two sections.

Python Pairwise IoU with broadcasting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The perfect guess scores 1.0 and the distant one 0.0. Moving a 40 by 40 box by only 10 units in each direction already drops IoU to 0.391, which shows how strict the measure is. Detection benchmarks usually call a prediction correct when its IoU with a true box is at least 0.5, and stricter evaluations average over thresholds from 0.5 up to 0.95.

You may wonder why detectors do not simply use mean squared error on the four coordinates. That error is not scale-aware: being 5 pixels off matters a lot for a 20-pixel object and almost nothing for a 400-pixel one. IoU is scale-free, so IoU-based losses (GIoU, DIoU, CIoU) are common in modern detectors.

Why duplicates appear, and non-maximum suppression

A detector makes predictions at many positions. A single dog covers dozens of neighboring cells, and each of them may produce a confident box around it. Left alone, the output would list the same dog many times.

Non-maximum suppression (NMS) removes the duplicates greedily:

  1. Sort all boxes of one class by score, highest first.
  2. Keep the top box.
  3. Discard every remaining box whose IoU with the kept box exceeds a threshold, typically 0.5.
  4. Repeat with the highest-scoring box that is left, until none remain.
Python Non-maximum suppression

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With thresholds 0.3, 0.5 and 0.7 the function keeps boxes 0, 3 and 5: one box per strong object, plus the weak box 5, which overlaps object A only a little and so survives. At 0.95 nothing counts as a duplicate any more, so all six boxes are returned (in score order: 0, 3, 1, 2, 4, 5). The threshold controls how much overlap you tolerate between two boxes you call different objects.

A low threshold is aggressive and might delete a real second object standing next to the first (a crowd of people, for example). A very high threshold keeps almost everything. Detectors run NMS separately for each class, and variants such as Soft-NMS lower a neighbor's score instead of deleting it. Some newer designs (DETR and its relatives) are trained to produce no duplicates, so they drop NMS entirely.

Two families of detectors

The first deep detectors reused a classifier and asked it many questions. They split into two schools.

Two-stage detectors first propose regions that might hold an object, then classify and refine each one.

  • R-CNN (2014) ran a classic algorithm to propose about 2,000 regions, cropped each one and ran a separate CNN on every crop. It was accurate but took many seconds per image, because the same pixels were processed again and again.
  • Fast R-CNN ran the CNN once on the whole image, then pooled a fixed-size feature patch for each proposed region (RoI pooling). The expensive part became shared.
  • Faster R-CNN replaced the hand-made proposal step with a small network, the region proposal network, that shares the same feature maps. The whole pipeline is trained end to end.

One-stage detectors skip proposals and predict boxes and classes directly from feature maps in a single pass.

  • YOLO ("You Only Look Once") divides the image into a grid. Every cell predicts boxes, an objectness score and class probabilities at once.
  • SSD predicts from feature maps at several resolutions, so large objects are found on coarse maps and small ones on fine maps.
  • RetinaNet introduced the focal loss, which down-weights the huge number of easy background predictions so they do not drown out the few real objects.
flowchart LR
    I["Image"] --> B["CNN backbone<br/>feature maps"]
    B --> R["Region proposals<br/>(two-stage only)"]
    R --> H1["Classify and refine<br/>each region"]
    B --> H2["Predict boxes and classes<br/>at every position<br/>(one-stage)"]
    H1 --> N["NMS"]
    H2 --> N
    N --> O["Final detections"]

Both families share a backbone; they differ in whether a separate proposal step comes before the final boxes.

Two-stage models were traditionally more accurate and one-stage models faster, but the gap has largely closed, and real-time one-stage detectors are the usual choice in products.

Anchors and matching

A network that predicts raw box coordinates from scratch is hard to train. Most detectors instead lay out anchor boxes (also called priors): a fixed set of boxes with several sizes and aspect ratios, centered on every position of a feature map. The network predicts offsets from the anchor rather than absolute coordinates, which is an easier regression because the starting guess is already roughly right.

During training you must decide which anchors are responsible for which true objects. The standard rule uses IoU:

  • An anchor whose IoU with some true box is at least about 0.5 is a positive: it should predict that object's class and box.
  • An anchor with low IoU against every true box (below roughly 0.3 to 0.4) is a negative: it should predict "background".
  • Anchors in between are ignored.
  • Each true box also claims its single best anchor, so that no object is left without a positive.

The standard offset encoding for a positive anchor with center (ax, ay) and size (aw, ah) and a true box (gx, gy, gw, gh) is:

TEXT
tx = (gx - ax) / aw       ty = (gy - ay) / ah
tw = log(gw / aw)         th = log(gh / ah)

Dividing by the anchor size makes the targets scale-free, and the logarithm keeps widths positive and treats "twice as wide" and "half as wide" symmetrically.

Python Anchor grid, matching and offset encoding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Of 96 anchors, only 4 are positive, 83 are clearly background and 9 are ignored. This heavy class imbalance is why focal loss and hard-negative mining exist. The tall object collects three positives (IoU 0.53 to 0.61), while the small one has only its single best anchor at IoU 0.39, which is exactly why the "best anchor" rule exists: without it that object would have no positive at all. The targets are small numbers near zero (the best anchor needs only a little shifting and rescaling), and decode inverts encode exactly.

In a real network, the regression head is trained to output these offsets for the positives (usually with a smooth L1 or IoU loss), the classification head learns object versus background for all non-ignored anchors, and at inference the decoded boxes go through NMS.

Newer detectors, called anchor-free (FCOS, CenterNet, YOLOv8 and later), drop the preset boxes and predict a distance to the four box edges from each location, but the matching idea stays: some positions are assigned to some objects, and a loss pulls them toward the truth.

Using a detector in practice

You rarely train a detector from scratch. You start from a model pretrained on a large dataset and fine-tune it, as described in Transfer Learning and Fine-Tuning. In PyTorch, the torchvision library ships ready-made detectors:

Python
# Read-only: PyTorch and torchvision are not available in the in-browser runner.
import torch
from torchvision.models.detection import fasterrcnn_resnet50_fpn, FasterRCNN_ResNet50_FPN_Weights

weights = FasterRCNN_ResNet50_FPN_Weights.DEFAULT
model = fasterrcnn_resnet50_fpn(weights=weights).eval()

image = torch.rand(3, 480, 640)               # replace with a real image tensor in [0, 1]
with torch.no_grad():
    result = model([image])[0]                # dict with "boxes", "labels", "scores"

keep = result["scores"] > 0.6
print(result["boxes"][keep], result["labels"][keep])

The model already applies NMS internally, so boxes, labels and scores are the final detections. To train on your own classes you replace the final predictor with one that has your class count and train with a list of {"boxes": ..., "labels": ...} targets, one dict per image.

Evaluation uses mean average precision (mAP). For each class, you sort the detections by score, mark each as a true or false positive using the IoU rule above, and compute the area under the precision-recall curve; mAP averages over classes (and often over several IoU thresholds). It rewards a detector that finds everything and ranks its correct boxes above its mistakes.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Two boxes overlap in an area of 200, and their areas are 400 and 300. What is their IoU?

    The union is 400 + 300 - 200 = 500, so IoU = 200 / 500 = 0.4. Dividing by the plain sum (700) would count the overlap twice.
  2. 2. What problem does non-maximum suppression solve?

    A single object triggers predictions at many nearby positions. NMS keeps the best-scoring box and removes neighbors that overlap it too much.
  3. 3. What is the main difference between two-stage and one-stage detectors?

    Faster R-CNN proposes regions and then classifies and refines them. YOLO and SSD predict classes and boxes straight from the feature maps.
  4. 4. Why does an anchor-based detector encode the width target as log(gw / aw)?

    The ratio gw / aw ignores the absolute size, and the logarithm turns multiplicative changes into additive ones. Decoding with exp always gives a positive width.

Summary

  • Object detection outputs a variable number of boxes, each with a class and a confidence score.
  • IoU, the shared area divided by the combined area, measures box agreement independent of scale; 0.5 is the usual "correct" threshold.
  • NMS sorts by score, keeps the best box and removes neighbors above an IoU threshold, per class.
  • Two-stage detectors (R-CNN family) propose and then refine regions; one-stage detectors (YOLO, SSD, RetinaNet) predict directly, and most are fast enough for real time.
  • Anchors are preset boxes the network adjusts by learned offsets; IoU matching decides positives and negatives, and the heavy imbalance motivates focal loss.
  • mAP summarizes detection quality by averaging precision over recall, classes and IoU thresholds.

Next: Image Segmentation.