Skip to content
Tutorials Center
My learning

Topics

Lesson 50 of 54 ยท Projects

Project: Panorama Stitching

Stitch two overlapping photos into a panorama: detect features, match them with a ratio test, fit a homography with RANSAC, then warp and blend. Live NumPy demo.

Panorama apps join several overlapping photos into one wide picture. The idea is elegant: if two photos share some of the same scene, we can find the same points in both, work out how the second photo must be bent to line up with the first, then paste the bent photo next to the first and hide the seam. Every step is a tool you have already met in this course: keypoints, descriptors, matching, homography, warping and masking.

The real pipeline uses cv2, which cannot run on this site, so it appears in read-only blocks. The live demos use a NumPy and Python stand-in on a synthetic scene: we cut two overlapping photos from a made-up wide image, pick keypoints, describe them with small patches, match with the ratio test, fit a homography with our own RANSAC loop, and warp and blend. The stand-ins are simpler than ORB or SIFT but follow the same logic, and the printed numbers come from real runs.

Stitching in five steps

Keypoints Find distinctive spots in both photos.
Descriptors Summarize each spot as numbers.
Matching Pair similar spots; keep the clear winners.
Homography RANSAC fits the warp, ignoring bad pairs.
1 / 2

The real pipeline (read-only)

Python
import cv2
import numpy as np

def stitch_pair(left, right):
    gray_l = cv2.cvtColor(left, cv2.COLOR_BGR2GRAY)
    gray_r = cv2.cvtColor(right, cv2.COLOR_BGR2GRAY)

    orb = cv2.ORB_create(nfeatures=2000)
    kp_l, des_l = orb.detectAndCompute(gray_l, None)
    kp_r, des_r = orb.detectAndCompute(gray_r, None)

    matcher = cv2.BFMatcher(cv2.NORM_HAMMING)
    pairs = matcher.knnMatch(des_r, des_l, k=2)                 # right -> left
    good = [m for m, n in pairs if m.distance < 0.75 * n.distance]
    if len(good) < 10:
        raise ValueError("not enough matches to stitch")

    src = np.float32([kp_r[m.queryIdx].pt for m in good]).reshape(-1, 1, 2)
    dst = np.float32([kp_l[m.trainIdx].pt for m in good]).reshape(-1, 1, 2)
    H, inliers = cv2.findHomography(src, dst, cv2.RANSAC, 5.0)

    height = max(left.shape[0], right.shape[0])
    width = left.shape[1] + right.shape[1]
    canvas = cv2.warpPerspective(right, H, (width, height))
    canvas[0:left.shape[0], 0:left.shape[1]] = left              # hard paste
    return canvas

# Or let OpenCV do everything, including blending and exposure compensation:
# stitcher = cv2.Stitcher_create()
# status, pano = stitcher.stitch([img1, img2, img3])

The functions are explained in Corners and Keypoints: Harris, Shi-Tomasi, ORB, Feature Matching and Homography and Perspective Transforms and Document Scanning. Notice we match right to left and fit a homography that maps right-photo coordinates into left-photo coordinates. That makes the left photo the reference frame, so it stays unwarped.

Real vs stand-in: ORB_create, BFMatcher, findHomography and warpPerspective are real OpenCV and read-only here, as is the all-in-one Stitcher. The live demos replace them with a local-maximum keypoint detector, normalized-patch descriptors, a brute-force matcher with the same ratio test, a hand-written RANSAC loop and a NumPy warp and blend.

Step 1: the photos, keypoints and descriptors

Our synthetic wide scene is 50 rows by 120 columns: a flat background with 250 random Gaussian blobs of different sizes and brightness (a stand-in for "interesting texture"). Photo A is the left part (columns 0 to 79). Photo B is the right part, taken a little higher: columns 45 to 119, starting at row 2. Each photo gets its own sensor noise, and photo B is 20 levels brighter, as if the camera auto-exposed differently. The true mapping from B coordinates to A coordinates is therefore a shift: x_A = x_B + 45 and y_A = y_B + 2.

For each photo we find keypoints (pixels that are brighter than all 8 neighbours and clearly above the background) and describe each one by a 9 by 9 patch of intensities, with its mean subtracted and its length normalized. Subtracting the mean and normalizing cancel brightness and contrast differences, so the same blob looks the same in both photos even with different exposure.

Python Keypoints and descriptors in two photos

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real detectors like ORB, SIFT and Harris-based methods are far more selective and are designed to survive rotation and scale changes. Our patch descriptor only copes with brightness changes and small noise, which is all this synthetic pair needs.

Step 2: matching with the ratio test

For each descriptor in B we search all descriptors in A and take the nearest and the second nearest. If the best match is much closer than the runner-up, it is probably right; if the two are similar, the match is ambiguous and likely wrong. This is Lowe's ratio test, and 0.75 is a common threshold. Many keypoints in B have no counterpart in A at all (they sit in the part of the scene A never saw), and the ratio test throws most of those out.

Step 3: fit the homography with RANSAC

Even after the ratio test, a few wrong matches survive. A plain least-squares fit would be dragged off by them. RANSAC (random sample consensus) handles this:

  1. Pick 4 random matches (the minimum that determines a homography).
  2. Solve for the 3x3 matrix.
  3. Count how many other matches agree with it, meaning the transformed point lands within a few pixels of its partner. These are the inliers.
  4. Repeat many times and keep the matrix with the most inliers.
  5. Refit using all the inliers for a final, accurate answer.

OpenCV's findHomography(src, dst, cv2.RANSAC, 5.0) does all of that, where 5.0 is the pixel tolerance in step 3. The demo below does it by hand and then compares the answer with the known shift.

Python Match, run RANSAC, recover the homography

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In our run, 14 matches survive the ratio test, RANSAC keeps 9 as inliers, and the matrix comes out as the identity with 45 and 2 in the last column: exactly the shift we built in. The 5 rejected matches are false ones that squeaked past the ratio test. We use a tolerance of 0.5 pixels because our synthetic keypoints are pixel-exact; with real photos you would use a few pixels, like the 5.0 in findHomography.

Step 4: warp and blend

Now we apply the homography to photo B and place it on a canvas wide enough for both photos. As in warpPerspective, we use inverse mapping: for each canvas pixel we ask where in B it comes from. In the overlap, both photos have data, and we have a choice:

  • Hard paste: pixels of A win. Fast, but any brightness difference makes a visible vertical seam.
  • Feathered blend: across the overlap, fade weights linearly from all-A to all-B. A brightness difference becomes a gentle ramp instead of a cliff.
  • Exposure compensation: scale B so its average matches A in the overlap before blending.

The demo prints one row of the canvas across the overlap for each method. Watch the jump at the seam in the hard paste, and the smooth ramp in the other two.

Python Warp, paste and feather across the overlap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The "true scene" row is what a perfect panorama would show. The hard paste matches it inside the overlap (photo A wins there) but jumps about 20 levels too bright at column 80, where photo A ends. The plain feather hides the jump but drifts toward B's brighter values across the whole overlap, so it is actually less accurate on average (mean error 10.0 against 7.6 for the hard paste). Compensating the gain first and then feathering brings the mean error down to 2.7. That is the lesson: blending hides seams, exposure compensation removes the cause. This is why real stitchers (including OpenCV's Stitcher) do exposure compensation and multi-band blending rather than a simple paste.

What breaks in real panoramas

  • Too little overlap. You need roughly 20 to 30 percent shared content for enough good matches.
  • Parallax. If the camera moves (rather than rotating about its center) near objects shift differently from far ones, and one homography cannot fit both. Rotate on the spot, or stitch distant scenes.
  • Repeating textures. Brick walls and tiled floors give ambiguous matches. The ratio test helps, but not always.
  • Moving objects. People walking between shots appear as ghosts; stitchers choose a seam that avoids them.
  • Lens distortion and vignetting. Calibrate and undistort first (Camera Calibration and Undistortion), otherwise lines bend across the seam.
  • Three or more photos. Chain the homographies, or let cv2.Stitcher estimate all cameras together and refine them (bundle adjustment).
  • Cylindrical or spherical warps. Wide panoramas stretch badly on a flat plane, so real ones project onto a cylinder or sphere first.

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does Lowe's ratio test check?

    If the nearest and second-nearest descriptors are about equally close, the match is ambiguous. Keeping only matches where the best is clearly better (for example below 0.75 times the second) removes many false pairs.
  2. 2. Why does findHomography use RANSAC instead of an ordinary least-squares fit on all matches?

    RANSAC fits many small random samples and keeps the model most matches agree with, so outliers do not influence it.
  3. 3. How many point pairs does the minimum RANSAC sample need to determine a homography?

    A homography has 8 degrees of freedom and each point pair gives 2 equations, so 4 pairs are the minimum.
  4. 4. In the blend step, why is a hard paste often visibly wrong even when the geometry is perfect?

    Cameras auto-expose each shot. Exposure compensation and feathered blending hide the difference.
  5. 5. A homography is fitted mapping right-photo points to left-photo points. Which photo stays unwarped?

    The matrix converts right-photo coordinates into the left photo's coordinate system, so the left photo already lives in the reference frame.

Summary

  • Panorama stitching: keypoints and descriptors in both photos, matching with a ratio test, a homography fitted with RANSAC, a warp onto a common canvas and a blend.
  • Fit the homography from the photo you want to move to the photo you keep as the reference.
  • RANSAC needs 4 point pairs per trial and keeps the model with the most inliers, which makes it robust to wrong matches.
  • Hard pastes show exposure seams; feathering and exposure compensation hide them.
  • Real panoramas need enough overlap, camera rotation without translation, and often cylindrical projection.
  • The live demos were NumPy stand-ins; the ORB and findHomography code is real but read-only here.

Next chapter: Practice Exercises with Solutions.