Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 54 ยท Detection with Machine Learning

QR Codes, ArUco Markers and Text

Detect and decode QR codes with QRCodeDetector, find ArUco markers and estimate distance, read text with a Tesseract bridge, and see how marker bits and IoU work.

Some things are easier to find when you design them to be found. A QR code carries a short piece of text. An ArUco marker is a small black-and-white square that gives a camera an ID and a precise position. Text in photos is the hard case, because letters are not designed for machines, and it needs OCR (optical character recognition).

OpenCV covers the first two directly and can feed the third to an OCR engine. This lesson shows the real calls (read-only, because opencv-python cannot run in this page) and then re-creates the interesting part, decoding marker bits and scoring detections, with plain Python in the live editors.

Three kinds of "reading"

QR code Carries text such as a URL.
ArUco marker Gives an ID and a pose.
OCR Turns pictures of text into strings.

The first two are built into OpenCV; OCR needs an extra engine.

1 / 2

QR codes

cv2.QRCodeDetector finds a QR code and decodes it in one call.

Python
import cv2

image = cv2.imread("ticket.png")
if image is None:
    raise SystemExit("could not read ticket.png")

detector = cv2.QRCodeDetector()
text, points, _ = detector.detectAndDecode(image)

if points is not None and text:
    print("payload:", text)
    corners = points.reshape(-1, 2).astype(int)   # four (x, y) corners
    for i in range(4):
        cv2.line(image, tuple(corners[i]), tuple(corners[(i + 1) % 4]), (0, 255, 0), 2)
else:
    print("no QR code decoded")

Three habits matter. First, the decoded text is an empty string when the code was found but could not be decoded, so test both points and text. Second, the corners come back in (x, y) order as floats with an extra axis, hence the reshape(-1, 2). Third, an image with several codes needs detectAndDecodeMulti, which returns lists.

QR codes include built-in error correction, so a code stays readable even when part of it is damaged or covered. The detector is also tolerant of some rotation and perspective, but blur, glare and tiny codes (under a few dozen pixels per module) still fail. When detection fails, try a larger image, better lighting, or a grayscale conversion first (see Color Spaces).

ArUco markers

An ArUco marker is a square with a thick black border and an inner grid of black and white cells. The grid encodes a number. A dictionary (for example DICT_4X4_50) defines how many cells there are and which patterns are valid, in this case a 4 by 4 grid and 50 distinct IDs.

Python
import cv2

dictionary = cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_4X4_50)
detector = cv2.aruco.ArucoDetector(dictionary, cv2.aruco.DetectorParameters())

gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
corners, ids, rejected = detector.detectMarkers(gray)

if ids is not None:
    cv2.aruco.drawDetectedMarkers(image, corners, ids)
    for marker_corners, marker_id in zip(corners, ids.flatten()):
        print(marker_id, marker_corners.reshape(4, 2))

ids is None when nothing was found, so check before using it. The rejected list holds square candidates that failed to decode, which is useful when debugging.

The ArUco functions live in the cv2.aruco module, which is part of the main package in OpenCV 4.7 and later. In older releases it was in the contrib build, and the function names were different (detectMarkers was a free function). If code from an old tutorial fails, check your version with cv2.__version__.

How the bits are decoded

The detector finds four corners, warps the marker to a flat square (a perspective transform), thresholds it, and samples one value per cell. Then it checks that the outer border is all black and matches the inner bits against the dictionary. Because a marker can be seen rotated, it tries all four rotations, and the one that matches tells you the marker's orientation as well as its ID.

Here is that last stage, working on an already flattened marker image.

Python Decode a marker grid

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The output shows an ID and a rotation: the rotated image reports 270 because the decoder had to turn the sampled grid three quarter-turns to match the dictionary. The broken marker is rejected rather than guessed, which is the right behavior. Real dictionaries are designed so that any two markers (in any rotation) differ in many cells, and OpenCV can even correct a bit or two of error. The two-entry dictionary above does no error correction.

Pose: how far away is the marker?

If you know the real size of a marker and the camera's focal length in pixels (from camera calibration), similar triangles give the distance. A marker S meters wide that appears p pixels wide is about f * S / p away. Full pose estimation with cv2.solvePnP uses all four corners to get rotation as well, but the one-number version builds intuition.

Python Distance from apparent size

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Halving the apparent width doubles the distance. That inverse relationship is the basis of both marker pose estimates and the stereo depth formula in Stereo Vision and Depth.

Reading text with OCR

OpenCV itself does not ship a general-purpose OCR engine for printed pages. The usual route is to use OpenCV to prepare the image and hand it to Tesseract, an open-source OCR engine, through the pytesseract package. Both the Tesseract program and the package must be installed on your machine, so this is read-only here.

Python
import cv2
import pytesseract

image = cv2.imread("receipt.jpg")
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
gray = cv2.resize(gray, None, fx=2, fy=2, interpolation=cv2.INTER_CUBIC)
_, binary = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)

text = pytesseract.image_to_string(binary, config="--psm 6")
print(text)

The preprocessing is what separates good from bad results: grayscale, upscale small text, binarize with Otsu thresholding, and straighten tilted pages with a perspective transform. OCR on photos of curved pages, handwriting or busy backgrounds is much less reliable than on clean scans, so always treat the output as a draft that needs validation.

OpenCV's dnn module can also run text-detection and text-recognition networks exported to ONNX, using the same blob workflow as in the dnn lesson.

A license plate reading pipeline

Reading a car's license plate is a good example of chaining the tools in this course. It is shown read-only, because it needs real photos and an OCR engine.

  1. Find the plate candidates. Convert to gray, smooth with a bilateral filter (details), find edges with Canny, then look for four-sided contours with approxPolyDP.
  2. Filter by shape. A plate is a rectangle with a known aspect ratio (about 4.7 to 1 for a standard European plate, 2 to 1 in the US), an area within a plausible range of the image, and a size of at least a few hundred pixels. Check w / h of the boundingRect against that ratio:
Python
def looks_like_plate(contour, image_area):
    x, y, w, h = cv2.boundingRect(contour)
    ratio = w / float(h)
    area_fraction = (w * h) / image_area
    return 2.0 < ratio < 6.0 and 0.002 < area_fraction < 0.1
  1. Straighten it. Warp the four corners to a flat rectangle with a perspective transform.
  2. Clean and segment. Threshold the crop (Otsu or adaptive), remove noise with morphology, and find the contours of the characters; keep those with a letter-like height.
  3. Recognize. Pass the binarized plate to an OCR engine as described above, with a whitelist of letters and digits, or classify each character with a small neural network.
  4. In video, run the pipeline on every n-th frame and vote: the plate text that appears most often across frames wins, which removes single-frame errors.

The weak points are glare, motion blur, night-time exposure and plates at steep angles. Production systems train a dedicated detector for the plate and use OCR designed for plates, and they handle personal data carefully because plate numbers are legally sensitive in many countries.

Scoring detections with IoU

How do you know whether a detector is any good? You compare its boxes to hand-drawn ground truth boxes with the IoU measure from the dnn lesson. A detection usually counts as correct when its IoU with an unmatched ground-truth box reaches 0.5. This works for text regions, markers, faces or anything else with a box.

Python Precision and recall from IoU

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Precision answers "of the things I reported, how many were real?" and recall answers "of the real things, how many did I find?". Raising a detector's confidence threshold typically raises precision and lowers recall, and you pick the balance your application needs.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. QRCodeDetector.detectAndDecode returns an empty string for the text. What can that mean?

    An empty string means no payload was decoded. Check the points too, and treat both as a normal 'not found' outcome.
  2. 2. Why does an ArUco decoder try four rotations of the inner bit grid?

    A square looks the same at any 90 degree turn, so the decoder tries each rotation and the one that matches gives both ID and orientation.
  3. 3. A 10 cm marker appears 80 px wide with a focal length of 800 px. Roughly how far away is it?

    distance = f * size / width_px = 800 * 0.10 / 80 = 1.0 meter.
  4. 4. Which step usually improves OCR results most?

    OCR engines work best on clean, high-contrast, upright text, which preprocessing helps to provide.

Summary

  • QRCodeDetector.detectAndDecode finds and reads a QR code; always check for an empty result.
  • ArUco markers give an ID and four corners; check that ids is not None, and mind the version differences of the cv2.aruco API.
  • Marker decoding is flatten, threshold, sample cells, check the border, then match four rotations against a dictionary.
  • Apparent size gives distance: f * S / p.
  • OCR uses an external engine such as Tesseract; OpenCV's job is preprocessing.
  • IoU compares detections to ground truth, from which you compute precision and recall.

Next, learn how cameras map the 3D world onto pixels in Camera Calibration and Undistortion.