Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 38 ยท Sensors and Perception

Cameras and Computer Vision

Understand how robot cameras work: the pinhole model, projecting 3D points, color blob detection, edge detection by convolution and depth from stereo, in NumPy.

Cameras are cheap, light and rich: one image holds color, texture, shape and, with some geometry, distance. That richness makes vision the hardest sensor to use. A camera does not hand you "cup at 0.4 m"; it gives you a grid of numbers, and software has to extract meaning. This lesson builds the basics from scratch with NumPy: how a camera maps the 3D world to pixels, how to find colored objects, how to detect edges, and how two cameras give depth.

The pinhole camera model

Imagine a tiny hole in a box: light from each point of the scene passes through it and lands on the back wall, forming an upside-down image. A real camera is a lens version of the same idea. For a 3D point (X, Y, Z) in the camera frame (Z pointing forward along the optical axis, X right, Y down), the pixel coordinates are

Text
u = fx ยท X / Z + cx
v = fy ยท Y / Z + cy

where fx, fy are the focal lengths in pixels and (cx, cy) is the principal point (usually near the image center). Dividing by Z is why far objects look smaller. These parameters form the intrinsic matrix K, found by calibration (photographing a checkerboard). A point first seen in the world must be moved into the camera frame by a homogeneous transform (the camera's extrinsics).

Python Projecting 3D points to pixels

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Doubling the distance halves the apparent size. You can turn this around: an object of known size shows its distance by its pixel size, a cheap trick for tracking a ball or a marker.

Images are arrays

A grayscale image is a 2D array of brightness values (0-255); a color image adds a third axis for red, green and blue. Vision code is therefore array code: slicing, comparing, averaging and convolving with NumPy or OpenCV.

Finding a colored object

A classic first robotics vision task: follow a red ball. Steps: threshold for the color, find the blob's pixels, compute its centroid and size, and steer to keep the centroid in the image center. Real code uses the HSV color space because hue is less sensitive to brightness than RGB; this example uses simple RGB rules on a synthetic image.

Python Detect a red ball and compute where to turn

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The size of the blob also estimates the distance: if the ball's real diameter is D, its pixel diameter d gives Z โ‰ˆ fx ยท D / d, directly from the pinhole equation.

Edges by convolution

An edge is a place where brightness changes quickly. Slide a small grid of weights (a kernel) over the image and sum the products at each position: this is convolution. The Sobel kernels respond to horizontal and vertical brightness gradients; the gradient magnitude highlights edges. Edges are the raw material for lane following, line tracking and shape detection (the Hough transform finds lines among edge points).

Python Sobel edge detection from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the trade-off between noise and detail: a lower threshold finds more edges but also noise. Smoothing the image first (a Gaussian blur) is the standard fix, as in the Canny detector.

Depth from stereo

Two cameras a distance B apart (the baseline) see the same point at slightly different horizontal pixels. The shift is the disparity d = uL - uR. By similar triangles,

Z = fx ยท B / d

so close objects have large disparity and far ones small. Matching pixels between the two images (stereo correspondence) is the hard part; the depth formula is easy. Because Z varies as 1/d, depth error grows rapidly with distance: a one-pixel matching error at 10 m is far worse than at 1 m. Other depth methods are structured light, time of flight cameras and monocular neural networks.

Python Depth from disparity and its error growth

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The modern vision pipeline

Today's robots mix classical geometry with learned models: neural networks detect and classify objects, estimate depth from one image and segment the scene, while geometric methods track motion and map (see localization and mapping). Whatever the method, treat vision output like any sensor reading: noisy, sometimes wrong, and best used together with other sensors.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In the pinhole model, what happens to an object's image size if it moves twice as far away?

    u = fx * X / Z, so doubling Z halves the pixel offsets.
  2. 2. Which parameters make up the camera intrinsics?

    Intrinsics describe the camera itself (fx, fy, cx, cy); extrinsics describe where it sits.
  3. 3. What does a larger stereo disparity indicate?

    Z = fx * B / d, so depth falls as disparity grows.
  4. 4. What does a Sobel kernel detect?

    Sobel kernels approximate horizontal and vertical derivatives of brightness.

Summary

  • The pinhole model maps 3D points to pixels with u = fx X/Z + cx; calibration finds the intrinsics.
  • Images are arrays; color thresholding plus a centroid is a complete ball-following pipeline.
  • Convolution with Sobel kernels highlights edges; smoothing reduces noise.
  • Stereo depth Z = fx B / d is accurate near the camera and degrades with distance.

Next: Sensor Fusion