Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 54 ยท Video and Cameras

Processing Video Frame by Frame

Build a fast OpenCV frame loop: resize for speed, time it with getTickCount, skip frames, and separate moving objects with MOG2, KNN or a running average.

Reading a video is easy; keeping up with it is the real work. A 30 fps stream gives you about 33 milliseconds per frame, and everything you do to a frame has to fit in that budget or your program falls behind. This lesson shows the usual frame-processing loop, the tricks that make it fast, and one of the most useful video techniques: background subtraction, which separates moving objects from a mostly static scene.

If you have not yet read frames from a file or camera, start with Reading and Writing Video.

Video decoding is not available in the browser, so the cv2 code is read-only. The live examples run on synthetic NumPy frames (a noisy gray background and a moving bright block) and reproduce the arithmetic of the techniques. Where OpenCV's own algorithm is more elaborate, the text says so.

The frame loop pattern

A good frame loop

Read Grab one frame, check ok.
Shrink Process a smaller copy.
Analyze Detect, track or subtract.
Show or save Draw results on the full frame.

Measure each stage. Guessing what is slow is usually wrong.

1 / 2
Python
import cv2

cap = cv2.VideoCapture("traffic.mp4")
frame_index = 0

while True:
    ok, frame = cap.read()
    if not ok:
        break
    frame_index += 1

    small = cv2.resize(frame, None, fx=0.5, fy=0.5, interpolation=cv2.INTER_AREA)
    gray = cv2.cvtColor(small, cv2.COLOR_BGR2GRAY)

    if frame_index % 3 == 0:          # heavy analysis on every third frame
        pass                           # run your detector here

    cv2.imshow("result", frame)
    if cv2.waitKey(1) & 0xFF == 27:    # Esc quits
        break

cap.release()
cv2.destroyAllWindows()

Shrinking is the cheapest big win, because most per-pixel work scales with the number of pixels: halving both sides leaves 25% of the work. Use INTER_AREA for shrinking (see Interpolation and Image Quality). If you detect something on the small frame, multiply its coordinates by the scale factor before drawing on the full-size frame.

Measuring time

Use OpenCV's tick counter to time a stage:

Python
t0 = cv2.getTickCount()
result = cv2.GaussianBlur(frame, (9, 9), 0)
t1 = cv2.getTickCount()

ms = (t1 - t0) / cv2.getTickFrequency() * 1000
print(f"blur took {ms:.2f} ms")

getTickCount() returns a raw counter and getTickFrequency() says how many ticks make one second. The number you get depends on your machine, so benchmark on the hardware you will deploy to. For the frame rate you actually achieve, time the whole loop and divide the frames processed by the seconds elapsed.

The tick arithmetic is simple enough to try without a camera. The stand-in below uses fake tick values (a made-up frequency) so the output is the same every time. In a real program the counter values come from cv2.getTickCount().

Python Tick arithmetic and the frame budget

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The analysis stage dominates, so shrinking the frame or running it less often helps far more than tuning the drawing code. Skipping frames only works when the task tolerates it: tracking a fast object needs every frame, counting cars crossing a line often does not.

Background subtraction

If the camera is fixed, the background (road, wall, room) stays almost the same and only people, cars and animals move. A background subtractor learns what the scene normally looks like and flags the pixels that differ. The result is a foreground mask: white where something moves, black elsewhere.

OpenCV ships two ready-made subtractors:

Python
subtractor = cv2.createBackgroundSubtractorMOG2(
    history=500,          # how many frames the model remembers
    varThreshold=16,      # how different a pixel must be to count as foreground
    detectShadows=True,   # shadows get gray value 127 in the mask
)
# or: cv2.createBackgroundSubtractorKNN(history=500, dist2Threshold=400.0)

while True:
    ok, frame = cap.read()
    if not ok:
        break
    mask = subtractor.apply(frame)               # uint8: 0, 127 (shadow), 255
    _, mask = cv2.threshold(mask, 200, 255, cv2.THRESH_BINARY)   # drop shadows
    mask = cv2.morphologyEx(mask, cv2.MORPH_OPEN, kernel)        # remove specks

MOG2 models each pixel as a mix of Gaussians and adapts to lighting changes and to the background shifting; KNN keeps recent samples and compares against them. Both are good defaults, MOG2 is a bit faster, and KNN can handle some scenes with many moving background parts better. After apply, the mask is usually noisy, so it is cleaned with thresholding to remove the shadow value and morphology (see Morphological Operations) to remove specks and fill holes.

Things to know:

  • The first frames are mostly "foreground" because the model has learned nothing yet. Give it a short warm-up.
  • A moving camera breaks the idea: everything changes, so everything is foreground.
  • An object that stops for long enough gets absorbed into the background (that is the adaptation at work).
  • learningRate in apply(frame, learningRate=...) sets how fast the model adapts: 0 freezes it, values near 1 forget the past almost instantly, and -1 (the default) lets OpenCV choose.

A running-average background in NumPy

The simplest background model is an exponential moving average of the frames:

TEXT
background = (1 - alpha) * background + alpha * frame

A pixel is foreground when it differs from the background by more than a threshold. This is not what MOG2 does inside (MOG2 keeps several Gaussians per pixel), but it shows the core idea of "learn the normal scene, flag the differences" and is available in OpenCV as cv2.accumulateWeighted.

The stand-in generates frames with a fixed seed: a gray background with a little noise, and a bright 4 by 4 block moving to the right.

Python Running-average background subtraction

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each printed mask shows the moving block as a cluster of #, but there is a second cluster on the left, at columns 2 to 5. That is a ghost: frame 0 already contained the block, so the background was initialized with it, and once the block moved away that spot looked "different" (the average slowly forgets it). Initializing from a frame without moving objects, or warming up for a while, avoids ghosts. Try a larger alpha to make the ghost fade sooner; the price is that a slow or stopped object also disappears into the background faster.

Putting it together

A typical motion-detecting loop combines everything above: resize, grayscale and blur to reduce noise, background-subtract, clean the mask, find the contours of the white regions (Finding and Drawing Contours), keep only contours above a minimum area, and draw their bounding boxes on the original frame. The motion detector project builds this step by step, and Motion and Object Tracking follows the detected objects over time.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is resizing frames before analysis such an effective speed-up?

    Most operations touch every pixel. Halving each side gives 25% of the pixels, so about a quarter of the work.
  2. 2. How do you convert cv2.getTickCount() differences into milliseconds?

    getTickFrequency() gives ticks per second. Dividing the tick difference by it gives seconds; times 1000 gives milliseconds.
  3. 3. What does a background subtractor such as MOG2 return from apply(frame)?

    apply returns a mask (255 for foreground, optionally 127 for shadows, 0 for background). You still clean it up and find contours yourself.
  4. 4. Which situation breaks background subtraction most clearly?

    The method assumes a mostly static background. With a moving camera every pixel changes, so everything looks like foreground.
  5. 5. In a running average, what happens when alpha is increased?

    alpha is the weight of the new frame. A large alpha forgets the past quickly, which helps with lighting changes but makes stopped or slow objects vanish into the background.

Summary

  • A frame loop has a time budget (about 33 ms at 30 fps); measure each stage with getTickCount and getTickFrequency.
  • Resize with INTER_AREA, use grayscale, restrict to an ROI and skip frames to stay within the budget.
  • Background subtraction (createBackgroundSubtractorMOG2 or KNN) learns a static scene and returns a foreground mask.
  • Clean the mask with a threshold (drop the shadow value 127) and morphological opening.
  • A running average (cv2.accumulateWeighted) is the simplest background model; alpha controls how fast it adapts.
  • Background subtraction needs a fixed camera and a short warm-up.

Next: Motion and Object Tracking