Skip to content
Tutorials Center
My learning

Topics

Lesson 42 of 54 ยท Camera Geometry

Stereo Vision and Depth

Understand stereo vision: epipolar geometry, disparity, depth = f times B over d, StereoBM and StereoSGBM in OpenCV, and block matching you can run on tiny arrays.

A single photo has no depth: a small object nearby and a big object far away can look identical. Your two eyes solve this by seeing the world from slightly different positions. Nearby things shift a lot between the two views, distant things shift very little, and your brain turns that shift into distance. Stereo vision does the same with two cameras.

In this lesson you will learn how the shift (the disparity) turns into depth, how OpenCV computes a disparity map with StereoBM and StereoSGBM, and how block matching works by running it yourself on a tiny pair of images.

What is real and what is a stand-in. The OpenCV calls are read-only here because opencv-python and stereo image pairs are not available in this page. The live examples implement the depth formula and a small block-matching algorithm in NumPy (real Pyodide). Real matchers are the same idea with faster code, better cost functions and smoothing.

Depth from two views

Two cameras Side by side, same height.
Disparity How far a point shifts between views.
Depth Big shift means near, small shift means far.

The whole method rests on finding the same point in both images.

1 / 2

The geometry

Put two identical cameras side by side, a distance B (the baseline) apart, both looking straight ahead. A point in the scene appears at horizontal pixel position xL in the left image and xR in the right image. The difference is the disparity:

Text
d = xL - xR      (in pixels, positive for points in front of the cameras)

Using similar triangles, the depth Z of the point is

Text
Z = f * B / d

where f is the focal length in pixels (from camera calibration). Three consequences are worth remembering. Depth is inversely proportional to disparity, so near objects have large disparities. A wider baseline gives larger disparities and so better precision at long range. And because disparity is measured in whole or fractional pixels, depth precision gets worse quickly with distance: at disparity 1 a one-pixel error is a huge depth error, while at disparity 40 it barely matters.

Python Disparity to depth

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A change of one pixel moves the 70 px disparity from 1.20 m to 1.22 m, but the 3 px disparity from 28 m to 42 m. Stereo is accurate close up and rough far away. That is why a robot with a 12 cm baseline can grasp a cup but cannot measure a distant building well.

Epipolar geometry and rectification

Where in the right image should you look for the match of a left-image pixel? Not everywhere. A point seen in one camera lies, in the other image, somewhere on a single line, the epipolar line. That follows from the geometry: the point, the two camera centers and the two image points all lie in one plane.

If the two cameras are perfectly parallel and level, the epipolar lines are exactly the image rows: a pixel in row 100 on the left can only match something in row 100 on the right. That turns a 2D search into a 1D one, which is the reason stereo is practical.

Real cameras are never perfectly aligned, so you apply stereo rectification: using calibration results, you warp both images so that the epipolar lines become horizontal and matching rows line up. OpenCV does this with cv2.stereoCalibrate (to get the relative pose of the two cameras), cv2.stereoRectify (to compute the transforms) and cv2.initUndistortRectifyMap plus cv2.remap (to warp the images).

The fundamental matrix

If you do not know the cameras' calibration, the epipolar geometry can still be estimated from point matches. The fundamental matrix F is a 3 by 3 matrix such that for every pair of matching points x (left) and x' (right) in homogeneous pixel coordinates, x'^T F x = 0. Multiplying F by a left point gives the epipolar line on which its match must lie in the right image. With calibrated cameras the same idea is the essential matrix E, which gives the rotation and translation between the cameras.

Python
pts1 = np.float32([k1[m.queryIdx].pt for m in good])
pts2 = np.float32([k2[m.trainIdx].pt for m in good])

F, inlier_mask = cv2.findFundamentalMat(pts1, pts2, cv2.FM_RANSAC, 1.0, 0.99)
pts1 = pts1[inlier_mask.ravel() == 1]
pts2 = pts2[inlier_mask.ravel() == 1]

lines_in_right = cv2.computeCorrespondEpilines(pts1.reshape(-1, 1, 2), 1, F).reshape(-1, 3)
# each line is (a, b, c): a*x + b*y + c = 0 in the right image

You need at least 8 matches for the 8-point method (RANSAC handles bad matches). Drawing the lines is a good sanity check: every matched point in the right image should sit on or very near its line, and the lines of a pair of cameras with a horizontal shift are nearly horizontal. Applications: filtering bad matches, rectifying uncalibrated images, tracking an object between views, and the first step of 3D reconstruction.

Computing a disparity map in OpenCV

Once images are rectified and converted to grayscale, a block matcher produces the disparity map.

Python
import cv2
import numpy as np

left = cv2.imread("left.png", cv2.IMREAD_GRAYSCALE)
right = cv2.imread("right.png", cv2.IMREAD_GRAYSCALE)

stereo = cv2.StereoSGBM_create(
    minDisparity=0,
    numDisparities=64,       # search range; must be divisible by 16
    blockSize=5,             # odd window size
    P1=8 * 5 * 5,            # smoothness penalties
    P2=32 * 5 * 5,
    uniquenessRatio=10,
)
disparity = stereo.compute(left, right).astype(np.float32) / 16.0   # fixed-point -> pixels

valid = disparity > 0
depth = np.zeros_like(disparity)
depth[valid] = focal_px * baseline_m / disparity[valid]

Notes that save hours:

  • StereoBM (simple block matching, fast, noisier) and StereoSGBM (semi-global matching, slower, smoother) share the same compute call. Start with SGBM for quality.
  • compute returns fixed-point values scaled by 16. Divide by 16 to get pixels.
  • numDisparities must be a multiple of 16 and sets how large a shift can be found. Too small and near objects fail; too large and it is slow.
  • Invalid or unmatched pixels get a negative value (or zero), so mask them before dividing.
  • Plain-colored walls, glossy surfaces and repeated patterns have no matchable texture, so they produce holes or errors.

For visualization, normalize the disparity to 0 to 255 and apply a color map. For real 3D points, cv2.reprojectImageTo3D(disparity, Q) uses the Q matrix from stereoRectify to return an (H, W, 3) array of coordinates.

Block matching, step by step

The idea behind StereoBM is simple. For each pixel in the left image, take a small window around it. Slide a window of the same size along the same row in the right image, over the range of allowed disparities, and compare each position with a cost such as the sum of absolute differences (SAD). The shift with the lowest cost is the disparity.

The example builds a left image of random texture, and a right image in which a square in the middle is shifted by 4 pixels while the background is shifted by 1 pixel (the square is "closer").

Python Block matching on a tiny pair

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Inside the square the estimate is 4 and in the background it is 1, so the two depth layers are recovered. Look closely at the square's edges: the estimated square spills about two columns past its true left edge, because windows there straddle two different depths and the foreground texture wins. This is the classic weakness of block matching, and the reason SGBM adds smoothness penalties. Also note that the loop starts max_disp pixels in from the left edge: a window cannot search past the image border, so real disparity maps have an unusable strip on one side.

From disparity to a depth map

With the disparity in hand, depth is one vectorized line. The example converts the estimate and masks out invalid values.

Python Depth map from disparity

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Using NaN for unknown depth keeps invalid pixels from silently looking like real measurements. Functions such as np.nanmin ignore them.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A point has disparity 20 px, the focal length is 600 px and the baseline is 0.1 m. What is its depth?

    Z = f * B / d = 600 * 0.1 / 20 = 3 meters.
  2. 2. Why is rectification useful?

    After rectification the epipolar lines are horizontal rows, which reduces the matching search to a 1D slide along a row.
  3. 3. StereoSGBM.compute returns values scaled by 16. What should you do before using them as pixel disparities?

    OpenCV stores disparities as fixed-point numbers with 4 fractional bits, so divide by 16.0 to get pixels.
  4. 4. Which situation usually breaks block matching?

    With no texture, every window looks the same, so there is no unique best match.

Summary

  • Disparity is the horizontal shift of a point between two views; depth is Z = f * B / d.
  • Depth is inversely proportional to disparity, so precision falls off with distance; a wider baseline helps.
  • Rectification aligns epipolar lines with image rows so matching is a 1D search.
  • StereoBM and StereoSGBM compute disparity; divide the fixed-point output by 16 and mask invalid pixels.
  • Block matching compares small windows (for example with SAD) over a range of shifts and keeps the best.

Next, make your code quicker in Making OpenCV Fast.