Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 54 ยท Features and Matching

Image Registration: Aligning Images with ECC and Phase Correlation

Align two images of the same scene: choose a motion model, estimate it with phase correlation, ECC or feature matching, warp the result, and fit an affine transform from point pairs.

Two photos of the same scene are rarely lined up. The camera moved a little between shots, a satellite passed over at a different angle, or a patient shifted between two scans. Image registration is the process of finding the geometric transformation that makes one image (the moving or input image) line up with another (the reference or template). Once aligned, you can compare them pixel by pixel, average them to reduce noise, merge exposures into an HDR image, subtract them to find changes, or stitch them into a panorama.

Registration has two parts: estimating the transformation and applying it with a warp. You have already met the second part in Translation, Rotation and Affine Transforms and Perspective Transforms and Document Scanning. This lesson focuses on the estimation methods OpenCV provides and when to pick each. The OpenCV calls are read-only; the NumPy models let you run phase correlation and an affine fit.

Step 1: choose a motion model

The model says what kinds of movement you allow. Simpler models are faster and more stable; use the simplest one that explains the difference between your images.

Model Degrees of freedom What it allows OpenCV constant
Translation 2 Shift in x and y cv2.MOTION_TRANSLATION
Euclidean (rigid) 3 Shift plus rotation cv2.MOTION_EUCLIDEAN
Affine 6 Also scale, shear cv2.MOTION_AFFINE
Homography 8 Perspective: a flat scene seen from a different viewpoint cv2.MOTION_HOMOGRAPHY

For example, two scans on a flatbed scanner need translation or rigid, two photos of a document from different angles need a homography, and a tripod time-lapse with tiny shakes needs translation.

Method 1: phase correlation (translation only)

If the images differ only by a shift, the Fourier transform makes the shift easy to find: a shift in space becomes a phase change in frequency. cv2.phaseCorrelate returns the shift with sub-pixel accuracy and is very fast. Both inputs must be single-channel float arrays of the same size.

Python
import cv2
import numpy as np

a = cv2.imread("frame1.png", cv2.IMREAD_GRAYSCALE).astype(np.float32)
b = cv2.imread("frame2.png", cv2.IMREAD_GRAYSCALE).astype(np.float32)

window = cv2.createHanningWindow(a.shape[::-1], cv2.CV_32F)   # reduces edge artifacts
(dx, dy), response = cv2.phaseCorrelate(a, b, window)
print(dx, dy, response)            # response near 1 means a confident match

It cannot handle rotation or scale, and a low response warns that the answer may be wrong.

Method 2: feature-based (ORB, SIFT) with RANSAC

This is the method behind panoramas: detect keypoints in both images, match their descriptors, and fit a transform to the matches while ignoring the wrong ones with RANSAC. It is robust to large movements, lighting changes and partial overlap.

Python
orb = cv2.ORB_create(2000)
k1, d1 = orb.detectAndCompute(ref_gray, None)
k2, d2 = orb.detectAndCompute(mov_gray, None)

matcher = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
matches = sorted(matcher.match(d2, d1), key=lambda m: m.distance)[:300]

src = np.float32([k2[m.queryIdx].pt for m in matches]).reshape(-1, 1, 2)
dst = np.float32([k1[m.trainIdx].pt for m in matches]).reshape(-1, 1, 2)

H, inliers = cv2.findHomography(src, dst, cv2.RANSAC, 5.0)
aligned = cv2.warpPerspective(mov, H, (ref.shape[1], ref.shape[0]))

For a camera that only slid and rotated (no perspective), cv2.estimateAffinePartial2D(src, dst) fits a similarity (shift, rotation, uniform scale) and cv2.estimateAffine2D fits a full affine. See Feature Matching and Homography for the details of matching and RANSAC.

Method 3: ECC, intensity-based alignment

The Enhanced Correlation Coefficient (ECC) method does not need keypoints. It searches for the warp that maximizes the correlation between the two images' pixel values. It gives very accurate results (sub-pixel) when the images are already close, and it is tolerant of brightness and contrast changes.

Python
ref = cv2.imread("reference.png", cv2.IMREAD_GRAYSCALE)
mov = cv2.imread("moving.png", cv2.IMREAD_GRAYSCALE)

warp = np.eye(2, 3, dtype=np.float32)                 # start with "no movement"
criteria = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 5000, 1e-6)

cc, warp = cv2.findTransformECC(ref, mov, warp, cv2.MOTION_AFFINE, criteria, None, 5)
print("correlation:", cc)

h, w = ref.shape
aligned = cv2.warpAffine(
    mov, warp, (w, h), flags=cv2.INTER_LINEAR + cv2.WARP_INVERSE_MAP
)

Key points:

  • For an affine, euclidean or translation model the matrix is 2 x 3 (np.eye(2, 3)); for MOTION_HOMOGRAPHY it is 3 x 3 (np.eye(3)), and you apply it with cv2.warpPerspective.
  • WARP_INVERSE_MAP is required. ECC finds the warp from the reference to the input, so you apply it in the inverse direction. Forgetting this flag is the most common bug.
  • The termination criteria stop after a number of iterations or when the improvement is tiny. If ECC raises "the algorithm did not converge", the images are too far apart; start closer.
  • The last argument (gaussFiltSize) blurs both images before comparing to reduce noise. The inputMask argument (here None) can exclude pixels such as a moving object.
  • It is an iterative local search, so it can get stuck. Fix this with a coarse-to-fine approach: first align heavily downsampled copies (see Image Pyramids), scale the result up, and refine on larger levels. Or initialize with a feature-based guess.

Which method?

Situation Best choice
Shift only, fast, video stabilization phaseCorrelate
Large motion, different viewpoint, partial overlap ORB or SIFT features + RANSAC homography
Small motion, accurate sub-pixel, color or lighting differences ECC
Different sensors (for example photo and map) Feature methods, or intensity methods designed for multi-modal images

Checking the result

An alignment can be wrong without any error message. Always verify:

  • Show the difference image: cv2.absdiff(ref, aligned). A good result is mostly black with thin edges.
  • Blend the two with cv2.addWeighted(ref, 0.5, aligned, 0.5, 0) and look for ghosting.
  • Use the numbers: the ECC correlation (close to 1 is good), the RANSAC inlier ratio, or the phase correlation response.
  • Compare the sum of absolute differences before and after; if it did not drop, the alignment failed.

Try it: phase correlation with NumPy's FFT

The model below creates a reference image with a distinctive pattern, shifts it by a known amount and recovers the shift with the same formula phaseCorrelate uses: normalize the cross-power spectrum, inverse-transform it and find the peak.

Python Find a shift with phase correlation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The estimate matches the true shift exactly and the difference drops to zero. An unrelated image gives a low peak, which is how you detect a failed registration.

Try it: fit an affine transform from point pairs

Feature-based methods end with a least-squares fit of a transformation to matched points. This model does that with np.linalg.lstsq for an affine model (the same math as cv2.estimateAffine2D without RANSAC), then uses it to align the points and measure the error, and shows how one wrong match damages the fit, which is why RANSAC exists.

Python Estimate an affine transform from matches

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What is real: the motion models, the OpenCV function names and argument order, the WARP_INVERSE_MAP requirement and the phase-correlation and least-squares mathematics are accurate. What is the stand-in: OpenCV cannot run in the browser, so NumPy re-creates phase correlation and the affine fit; real findTransformECC and RANSAC are iterative algorithms and more elaborate than these sketches.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which method finds a pure translation between two images quickly using the Fourier transform?

    A spatial shift becomes a phase difference in the frequency domain, so phaseCorrelate recovers (dx, dy) with sub-pixel accuracy.
  2. 2. What flag must you add when applying the warp that cv2.findTransformECC returns?

    ECC estimates the warp from the reference to the input, so you must apply it in inverse mode with WARP_INVERSE_MAP.
  3. 3. What shape should the warp matrix be for MOTION_AFFINE?

    Translation, euclidean and affine use a 2x3 matrix (np.eye(2, 3, dtype=np.float32)); homography uses 3x3.
  4. 4. Why is RANSAC used when fitting a transform to feature matches?

    A few false matches can ruin a plain fit. RANSAC repeatedly fits to random subsets and keeps the model that most matches agree with.
  5. 5. How can you check that an alignment worked?

    Alignment can fail silently, so inspect the difference image and the quality numbers.

Summary

  • Registration finds the transform that aligns a moving image to a reference; then you apply it with warpAffine or warpPerspective.
  • Pick the simplest motion model: translation (2 DOF), euclidean (3), affine (6) or homography (8).
  • phaseCorrelate is fast for pure shifts; ORB or SIFT plus RANSAC (findHomography, estimateAffine2D) handles large motion; findTransformECC gives accurate intensity-based alignment for small differences.
  • ECC needs a starting matrix, termination criteria and WARP_INVERSE_MAP when applying the result, and works best coarse-to-fine.
  • Always verify with a difference image and the quality numbers.

Next: look at the geometry of two cameras in Camera Calibration and Undistortion.