Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 46 ยท Classical Machine Learning for Images

Hand-Crafted Features and HOG

Why raw pixels make poor features, how gradient-orientation histograms and HOG describe shape, and how color histograms help. Built from scratch in NumPy.

A classifier needs numbers, so the first idea is to hand it the pixels: flatten the image into a long vector and let the model sort it out. That works for tiny, tidy pictures and fails quickly on anything else. This lesson shows why, and introduces the classic fix: compute features that keep what matters (shape, texture, color distribution) and discard what does not (exact position, small lighting changes).

The most famous hand-made shape feature is HOG, the Histogram of Oriented Gradients. It powered pedestrian detectors for years and is still a strong, fast baseline on small data. You will build it from scratch, using the gradients from Gradients and Edge Detection, and then add the simplest appearance feature, the color histogram.

Why raw pixels fail

Distance between two flattened images is the usual way a classifier or a search engine decides "similar". But pixel distance is extremely sensitive to small shifts. Move an object by one pixel and nearly every pixel that mattered changes.

The example makes this concrete with a thin vertical bar. Compare it with the same bar moved one pixel, the bar moved six pixels, and a horizontal bar of the same length. Then repeat with a simple gradient-orientation histogram: a count of how many gradient pixels point in each direction, weighted by edge strength.

Python Pixels versus gradient directions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In pixel space the bar is exactly as far from its one-pixel-shifted copy (4.0) as from the copy six pixels away (4.0), because in both cases no bright pixels overlap. Even worse, the vertical bar is closer to the horizontal one (3.74) than to its own shifted twin, because the two bars cross and share a pixel. A nearest-neighbor classifier on raw pixels would call the horizontal bar the better match.

The gradient-direction histograms tell the right story. The vertical bar and its shifted copies have identical histograms (distance 0.0): most of the edge strength (0.8) points sideways, with a smaller part (0.2) coming from the bar's two ends, whose gradients point up and down. The horizontal bar has the opposite split, so the distance to it is large (0.85). The feature has thrown away where and kept what direction the edges run.

Gradients as the raw material

At every pixel, the gradient has a magnitude (how fast brightness changes) and a direction (which way it increases). A flat region has zero magnitude. An edge has a large magnitude and a direction perpendicular to the edge: a vertical edge has a horizontal gradient.

Two choices matter for HOG:

  • Unsigned orientation. A dark-to-light edge and a light-to-dark edge have gradients pointing in opposite directions (0 and 180 degrees). For shape, both are "a vertical edge", so we fold the angle into 0 to 180 degrees with % 180.
  • Magnitude weighting. Strong edges should count more than faint noise, so each pixel votes with its magnitude rather than a count of one.

The HOG recipe

HOG summarizes an image in four steps.

  1. Compute the gradient magnitude and orientation at every pixel.
  2. Divide the image into small cells (for example 4 by 4 pixels) and build one histogram of orientations per cell, usually with 9 bins of 20 degrees. Each pixel splits its vote between the two nearest bins so that a tiny angle change does not make the vote jump.
  3. Group cells into overlapping blocks (for example 2 by 2 cells) and normalize each block's concatenated histograms to unit length. This makes the feature robust to local contrast and lighting changes.
  4. Concatenate all block vectors into one long feature vector.
flowchart LR
    A["Image"] --> B["Gradients<br/>magnitude, angle"]
    B --> C["Cells (4x4 px)<br/>9-bin histograms"]
    C --> D["Blocks (2x2 cells)<br/>normalized"]
    D --> E["One long vector<br/>= HOG feature"]

HOG keeps local edge directions, loses exact positions inside a cell, and normalizes away lighting.

The idea behind the cells is a deliberate loss of precision: the histogram says how much of each direction exists in a small neighborhood but not exactly where. This is why a shift of a pixel or two barely changes the descriptor. Overlapping block normalization is the other trick: each cell appears in several blocks, each normalized differently, which makes the feature steadier.

HOG from scratch

Here is a complete implementation for small grayscale images, with soft binning between neighboring orientation bins and block normalization. It uses three 16 by 16 test images: a vertical, a horizontal and a diagonal edge.

Python A HOG descriptor in NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A 16 by 16 image with 4 by 4 cells has 4 by 4 cells, hence 3 by 3 = 9 overlapping blocks. Each block has 2 x 2 x 9 = 36 numbers, so the descriptor has 324 values. Summing them over all blocks gives a rough orientation profile of the image:

  • The vertical edge has its energy in the first and last bins (0.5 each). Its gradient points at exactly 0 degrees, which lies on the border between the bin centered at 170 and the one centered at 10, so the vote is split evenly. Orientation wraps around, which is why lo % bins is in the code.
  • The horizontal edge puts everything in the bin centered at 90 degrees.
  • The diagonal edge has gradients near 135 degrees, which falls between the 130 and 150 degree bins: 0.71 and 0.25.

The last two lines are the payoff. Moving the edge by one pixel changes 16 pixels, a pixel distance of 4.0, but the HOG distance is 0.0 because both positions lie in the same cell. Comparing the vertical edge with the horizontal edge gives a HOG distance of about 3.87. The descriptor separates different shapes and forgives small shifts.

Note: Library implementations (for example the one in scikit-image) add options such as signed gradients, other block shapes and different normalizations. The classic pedestrian detector used 64 by 128 pixel windows with 8 by 8 pixel cells; for tiny 8 by 8 digit images you would use much smaller cells, such as 2 by 2 pixels.

Color histograms as features

HOG throws away color. Sometimes color is exactly what you want: telling sunsets from forests, ripe fruit from unripe fruit, or finding photos with similar palettes. A color histogram counts how many pixels fall into each color bin, ignoring where they are. The common recipe is one histogram per channel (for example 4 to 16 bins), each divided by the pixel count so that images of different sizes are comparable, then joined into one vector.

Like HOG, it trades position for robustness. The example paints a red block on a light background, placed on the left, on the right or in the middle, and a blue block on the left. Pixel distance and histogram distance are compared.

Python Color histograms ignore position

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With raw pixels the red block on the left looks more different from the red block on the right (2795) than from a blue block on the left (1689). With 4-bin histograms the order is reversed: the two red images are almost identical (distance 0.023 and 0.016 for the centered one), while the red-versus-blue distance is 0.75. The feature vector has only 12 numbers, four per channel.

A color histogram has its own weaknesses. It cannot tell a red car on a gray road from a gray car on a red road, since position is gone. It reacts to lighting (a shadow moves pixels to darker bins), which is why histograms are often computed in HSV or Lab space where lightness is separate (see Color Channels and Color Spaces), and it depends on how much of the image is background. Choosing the number of bins is a trade-off: too few bins blur real differences, too many make nearly identical images look different.

Combining features

Hand-crafted features are building blocks, not rivals. A practical feature vector for a small classifier is often the concatenation of several kinds: HOG for shape, a color histogram for palette, maybe a texture statistic. Two practical notes:

  • Scale the parts. A block of 324 HOG values and a block of 12 histogram values should not dominate each other just because of length or units. Standardize features (zero mean, unit variance) before a distance-based or linear model.
  • Keep the pipeline identical at training and prediction time. Every parameter (cell size, bins, normalization) is part of the model.

In the next lesson, Classifying Handwritten Digits, you will compare classifiers fed with raw pixels against classifiers fed with HOG features.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In the bar example, why was the vertical bar closer (in raw pixel distance) to a horizontal bar than to its own copy moved by one pixel?

    Pixel distance counts mismatched pixels. The crossing bars overlap in one pixel, so they mismatch in slightly fewer places than a shifted copy that overlaps nowhere. It says nothing about shape.
  2. 2. Why does HOG fold gradient angles into the range 0 to 180 degrees ('unsigned' orientation)?

    The sign of the gradient depends on which side is brighter. Folding by 180 degrees makes the descriptor describe the edge direction regardless of which side is lighter.
  3. 3. A 16 x 16 image, 4 x 4 pixel cells, 9 bins and 2 x 2 cell blocks with overlap of one cell. How many numbers are in the HOG vector?

    There are 4 x 4 cells, so 3 x 3 = 9 overlapping blocks. Each block holds 2 x 2 x 9 = 36 values, and 9 x 36 = 324.
  4. 4. What is the main reason for normalizing HOG over blocks?

    Dividing each block's histogram by its length removes the overall strength of the edges in that neighborhood, so a faint copy and a bright copy of the same shape give similar features.
  5. 5. Which task is a color histogram a poor feature for?

    A histogram counts colors but forgets where they are, so both pictures contain the same mix of red and gray. Normalizing by pixel count already makes different sizes comparable.

Summary

  • Raw pixel vectors are fragile: a one-pixel shift can change the distance more than a change of shape does.
  • Features keep the useful information and drop the nuisance. HOG keeps local edge orientations; color histograms keep the palette.
  • HOG recipe: gradients, orientation histograms per small cell (soft-binned, unsigned, magnitude-weighted), overlapping block normalization, one long vector.
  • A 16 by 16 image with 4 by 4 cells and 9 bins gives 3 x 3 blocks of 36 values each, 324 in total.
  • Color histograms are position-free and cheap but depend on lighting and background share. Combine features and standardize them before modeling.

Next, put the features to work: Classifying Handwritten Digits.