Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 54 ยท Video and Cameras

Motion and Object Tracking

Track objects in OpenCV with color masks, Lucas-Kanade optical flow, meanShift and CamShift, and CSRT or KCF trackers. Live NumPy flow and centroid tracker.

Detecting an object in one frame is a different problem from tracking it across many. Tracking has to keep the same identity from frame to frame, and it can use what it already knows: the object was just over there, so look nearby. That makes tracking faster than detection and lets it cope with moments when detection would fail.

This lesson covers four families of techniques, from the simplest to the most capable: color tracking, optical flow, mean-shift style tracking and the trainable trackers (CSRT, KCF). It ends with a tiny centroid tracker that gives detected objects stable IDs, which is how many real systems combine a detector with tracking.

Cameras and video files are not available in the browser, so the cv2 code is read-only. The live examples run on synthetic NumPy images and implement the underlying math (a color mask and centroid, one Lucas-Kanade step, and a centroid tracker) so you can see how the ideas work. They are stand-ins, not OpenCV's own implementations.

Four ways to follow something

Color Mask a color, take the centroid.
Optical flow Follow points from frame to frame.
meanShift Climb to the densest nearby spot.
CSRT and KCF Learn the object's look and re-find it.

Pick the simplest one that works for your scene.

1 / 2

Color tracking with inRange

If the object has a distinctive color (a red ball, a green marker), you can threshold the color and take the centroid of the white pixels. Work in HSV so that lighting changes affect V more than H (see Color Spaces: BGR, RGB, Gray and HSV):

Python
import cv2
import numpy as np

cap = cv2.VideoCapture(0)
lower = np.array([35, 100, 100])     # green-ish in OpenCV's HSV (H is 0 to 179)
upper = np.array([85, 255, 255])

while True:
    ok, frame = cap.read()
    if not ok:
        break
    hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
    mask = cv2.inRange(hsv, lower, upper)
    mask = cv2.erode(mask, None, iterations=2)
    mask = cv2.dilate(mask, None, iterations=2)

    m = cv2.moments(mask, binaryImage=True)
    if m["m00"] > 0:
        cx = int(m["m10"] / m["m00"])
        cy = int(m["m01"] / m["m00"])
        cv2.circle(frame, (cx, cy), 8, (0, 0, 255), -1)

inRange produces a 0/255 mask, the erode and dilate pair removes specks, and the centroid comes from image moments: m10 / m00 is the mean x of the white pixels and m01 / m00 the mean y. Check m00 > 0 first, otherwise the object is out of view and you would divide by zero. This approach is quick and surprisingly robust, but fails when other things share the color or the lighting changes the hue.

The stand-in builds a synthetic color image with a red ball that moves, masks it with a range test (on BGR for simplicity, since our synthetic colors are exact) and computes the centroid like the moments formula.

Python Color mask and centroid

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The centroid follows the ball by 6 columns per frame, and the green distractor is ignored because it fails the range test. Red in a real camera image would span a range in HSV hue, wrapping around 0 and 179, so real code often combines two inRange masks for red.

Optical flow: where did each point go?

Optical flow estimates how pixels move between two frames. The Lucas-Kanade method tracks a sparse set of points. It assumes that within a small window all pixels move the same way and that brightness does not change, which gives one equation per pixel in the window:

TEXT
Ix * u + Iy * v = -It

where Ix and Iy are the spatial gradients, It is the change over time and (u, v) is the motion to solve for. With many pixels in the window the system is overdetermined and is solved by least squares. OpenCV runs this on an image pyramid so that larger motions work too:

Python
old_gray = cv2.cvtColor(old_frame, cv2.COLOR_BGR2GRAY)
p0 = cv2.goodFeaturesToTrack(old_gray, maxCorners=100, qualityLevel=0.3, minDistance=7)

lk = dict(winSize=(15, 15), maxLevel=2,
          criteria=(cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03))

while True:
    ok, frame = cap.read()
    if not ok:
        break
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    p1, status, err = cv2.calcOpticalFlowPyrLK(old_gray, gray, p0, None, **lk)

    good_new = p1[status.flatten() == 1]
    good_old = p0[status.flatten() == 1]

    old_gray = gray
    p0 = good_new.reshape(-1, 1, 2)

The good points to follow come from goodFeaturesToTrack (see Corners and Keypoints). status marks which points were found again; drop the others. Points drift and get lost over time, so real code re-detects corners now and then. For flow at every pixel, cv2.calcOpticalFlowFarneback returns a dense (H, W, 2) field.

Here is one Lucas-Kanade step in NumPy. A smooth blob moves one pixel right and half a pixel down; we estimate (u, v) for a window around it by solving the least-squares system.

Python One Lucas-Kanade step

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The estimate matches the true motion here because the shift is small and the blob is smooth. Lucas-Kanade is a first-order (linearized) method, so it gets less accurate as motions grow, which is why OpenCV runs it on a pyramid for larger ones. The last two lines print the smaller singular value of the system, a measure of how well both motion components are determined. In the flat corner of the image it is orders of magnitude smaller, so the solution would be dominated by noise: this is the aperture problem in its extreme form, and the reason you track corners instead of flat or straight-edge regions.

meanShift and CamShift

meanShift tracks a region by repeatedly moving a window to the center of mass of a probability map inside it, until it stops moving. The usual probability map is a back-projection of a color histogram: how likely each pixel is to belong to the object, based on a histogram of the object's hue. CamShift (Continuously Adaptive meanShift) additionally resizes and rotates the window.

Python
x, y, w, h = 300, 200, 80, 80                      # initial window, e.g. selected by hand
roi = frame[y:y + h, x:x + w]
hsv_roi = cv2.cvtColor(roi, cv2.COLOR_BGR2HSV)
mask = cv2.inRange(hsv_roi, (0, 60, 32), (180, 255, 255))
hist = cv2.calcHist([hsv_roi], [0], mask, [180], [0, 180])
cv2.normalize(hist, hist, 0, 255, cv2.NORM_MINMAX)

term = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 1)
track_window = (x, y, w, h)

# for each new frame:
hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
back = cv2.calcBackProject([hsv], [0], hist, [0, 180], 1)
rot_rect, track_window = cv2.CamShift(back, track_window, term)
# or: _, track_window = cv2.meanShift(back, track_window, term)

These are cheap and need no training, but they depend on color, so a similarly colored background confuses them, and they need a good initial window.

Trackers: CSRT and KCF

OpenCV's tracking API takes an initial bounding box and then, for each frame, returns the object's new box. KCF is fast and accurate in easy cases; CSRT is slower but more accurate and copes better with changes of scale and partial occlusion. Which trackers exist depends on your build: in recent versions CSRT and KCF are in the main cv2 namespace, in older ones you needed opencv-contrib-python and names like cv2.TrackerCSRT_create. Check what your install provides.

Python
tracker = cv2.TrackerCSRT_create()                   # or cv2.TrackerKCF_create()
ok, frame = cap.read()
bbox = cv2.selectROI("select", frame, False)         # needs a GUI
tracker.init(frame, bbox)

while True:
    ok, frame = cap.read()
    if not ok:
        break
    found, bbox = tracker.update(frame)
    if found:
        x, y, w, h = map(int, bbox)
        cv2.rectangle(frame, (x, y), (x + w, y + h), (0, 255, 0), 2)
    else:
        cv2.putText(frame, "lost", (20, 40), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 0, 255), 2)

A single tracker follows a single object; for several objects you create one tracker per object. Trackers do not know what an object is, so they can slide onto the background when the object is lost and still report found = True for a while. Running a detector regularly and re-initializing the trackers is the standard remedy.

A centroid tracker in plain Python

When a detector gives you boxes in every frame, you still need to know which box in this frame is which box in the last. A centroid tracker does it in the simplest way: match each new detection to the nearest existing track, create a new track for unmatched detections, and forget tracks that have not been seen for a few frames.

Python A centroid tracker with stable IDs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first two objects keep IDs 0 and 1 even though the detector misses object 1 for one frame, because the track survives max_missing frames, and the new object gets the next free ID 2. Real systems add motion prediction (a Kalman filter) and better assignment (the Hungarian algorithm), but the structure is the same.

Choosing a method

  • A single, distinctively colored object on a controlled background: color tracking (fast, easy).
  • Following a set of corner points between frames, for example for motion estimation: Lucas-Kanade.
  • One object you can select once, with the camera running: CSRT for accuracy, KCF for speed.
  • Several objects of a known kind: a detector (see Face Detection with Haar Cascades or the dnn module) plus a centroid tracker.
  • Fixed camera and only motion matters: background subtraction.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In color tracking, how is the centroid of a binary mask computed from image moments?

    m00 is the area of the white region, m10 the sum of x coordinates and m01 the sum of y coordinates. Dividing gives the mean position.
  2. 2. What does Lucas-Kanade optical flow assume?

    Brightness constancy plus a locally uniform motion gives one linear equation per pixel in the window, solved by least squares.
  3. 3. Why do we track corners rather than pixels in flat regions?

    Without gradients in two directions, the Lucas-Kanade system is under-determined. Corners provide gradients both ways.
  4. 4. Which statement about CSRT and KCF is correct?

    Trackers are initialized with one box and follow that one object. KCF favors speed, CSRT accuracy.
  5. 5. What is the purpose of a centroid tracker on top of a detector?

    A detector alone returns unlabeled boxes per frame. The centroid tracker links them over time with stable IDs.

Summary

  • Tracking uses the previous position to follow an object; detection finds objects from scratch.
  • Color tracking is inRange plus morphology plus a centroid from moments, and works only if the color is distinctive.
  • Lucas-Kanade (calcOpticalFlowPyrLK) follows corners from frame to frame; flat areas and straight edges cannot be tracked.
  • meanShift and CamShift climb a back-projected color histogram.
  • CSRT and KCF follow one object from an initial box; availability depends on your OpenCV build.
  • A centroid tracker gives detections stable IDs by matching nearest centroids and tolerating short misses.

Next: Face Detection with Haar Cascades