Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 46 ยท Classical Machine Learning for Images

PCA, Eigenfaces and Image Compression

Use principal component analysis on images: flatten, fit PCA with scikit-learn, read eigen-images, measure reconstruction error, and compress with SVD.

Images are high-dimensional. A modest 64 by 64 photo has 4,096 pixels, yet pictures of the same kind of thing (faces, digits, handwriting, product shots) are far from random. Neighboring pixels move together, and a few patterns explain most of the variation. Principal component analysis (PCA) finds those patterns automatically. It rotates the data into new axes, ordered by how much of the variation each carries, so you can keep the first few and drop the rest.

That one idea gives you three tools. You can compress images by storing only a few numbers per picture, denoise them by discarding the weak axes, and build compact features for a classifier. The most famous application is eigenfaces: the principal components of a face dataset, displayed as images. In this lesson you will run PCA on the digits, measure what you lose when you compress, build a miniature eigenface example from generated faces, and compress a single picture with the singular value decomposition (SVD). If linear algebra feels distant, Eigenvalues, SVD and Decompositions is a good companion.

Flattening images into vectors

PCA works on a table: one row per sample, one column per feature. For images the sample is a picture and the features are its pixels, so each image of shape (H, W) is flattened into a row of length H * W. A dataset of 1,797 digits of 8 by 8 pixels becomes a matrix of shape (1797, 64).

Each row is a point in a 64-dimensional space. PCA looks at the cloud of points and does three things:

  1. It subtracts the mean image from every row, so the cloud is centered at the origin.
  2. It finds the direction of greatest spread, the first principal component.
  3. It finds the next direction, perpendicular to the first, with the most remaining spread, and so on.

Each component is a vector of 64 numbers, so it can be reshaped back to 8 by 8 and viewed as an image. These are the eigen-images. Every picture in the dataset is approximated as the mean image plus a weighted sum of a few eigen-images:

image โ‰ˆ mean + w1 * component1 + w2 * component2 + ... + wk * componentk

The weights w1 ... wk are the compact description of the image.

flowchart LR
    A["Images<br/>(N, H, W)"] --> B["Flatten<br/>(N, H*W)"]
    B --> C["Subtract mean,<br/>fit PCA"]
    C --> D["Keep k components<br/>(N, k) weights"]
    D --> E["Reconstruct<br/>mean + weights x components"]
    E --> F["Reshape<br/>(N, H, W)"]

PCA turns each image into k weights and can rebuild an approximation from them.

PCA on the digits

scikit-learn's PCA does the fitting. After fit, three attributes matter: mean_ (the mean image), components_ (one row per principal component, each of length H * W) and explained_variance_ratio_ (the share of total variance each component carries, in decreasing order).

Python How many components do the digits need?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first component alone explains 14.9 percent of the variance, the second 13.6 percent and the third 11.8 percent. Variance accumulates slowly: 13 components are needed for 80 percent, 21 for 90 percent, 29 for 95 percent and 41 for 99 percent of the original 64. Handwritten digits are not a tiny family of shapes, so the compression is real but moderate.

The mean digit looks like a blurry blob that mixes all ten classes. The principal components look nothing like digits. They are patterns of plus and minus: wherever the component is bright, the image gets brighter when the weight is positive, and where it is dark, the image gets darker. They are differences from the mean, which is why they are best viewed with the display scaled between their minimum and maximum, as the helper does.

Note that you read explained_variance_ratio_ to choose k. A common rule is "keep enough components to explain 90 to 99 percent of the variance". You can also pass a fraction directly, as in PCA(n_components=0.95).

Reconstruction error

What do you lose by keeping only k components? Project an image to k weights with transform, rebuild it with inverse_transform, and compare the result with the original. The mean squared difference is the reconstruction error.

To be honest about it, the PCA below is fitted on one set of images and the error is measured on 497 other images that were not part of the fit.

Python Reconstruction error versus components

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The error shrinks steadily: 0.053 with 2 components, 0.020 with 10, 0.008 with 20, 0.0009 with 40, and exactly 0 with all 64 (a rotation loses nothing). The pictures show what the numbers mean. With 2 components the reconstruction is a smeared blob. With 10 the loop and the stem of the 9 are clear but fuzzy. With 40 the reconstruction is nearly indistinguishable from the original.

This trade-off is the heart of lossy compression: quality against the number of stored values. It is also a reason PCA can help denoise: noise spreads evenly over all directions, while the signal sits in the first components, so reconstructing from the leading ones throws away most of the noise.

Eigenfaces

The classic demonstration is a set of aligned, centered face photos. The mean image is the "average face". The first eigenface is the dominant pattern of variation, usually related to overall lighting or head shape, and the following ones capture things like glasses, beards and expression. Any face is approximated as the average face plus a weighted sum of a few dozen eigenfaces, and recognition can compare the short weight vectors instead of the pictures.

Real face data cannot be downloaded in this in-browser environment, so here is a stand-in that shows the same effect honestly: 300 generated 16 by 16 "faces" built from just three hidden factors (head width, eye spacing and mouth height) plus a little noise. A good PCA should discover that the 256 pixels really depend on a few things.

Python Eigenfaces from generated faces

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Out of 256 pixels, only 12 components are needed for 95 percent of the variance. The first one alone carries 34.7 percent, five carry 81.3 percent and ten carry 94.4 percent. The mean face is a smooth ellipse with eyes and a mouth, the picture of "an average face". The first eigenface is a ring around the edge of the head: it is the pattern that makes the head wider or narrower, and the weight of that component correlates with the true head width at 0.98. PCA was never told about head width; it found the strongest direction of variation and that turned out to be it.

Real eigenfaces work in the same way but with caveats you should remember. They need aligned faces (the eyes in the same place), and lighting often becomes the first and strongest component, which is a nuisance for recognition. Today deep networks that learn face embeddings have replaced eigenfaces in practice, yet PCA remains a standard first step for visualizing and compressing image features.

Compressing one image with SVD

PCA compresses a collection. To compress a single picture you can apply the same math to the pixel matrix itself. The singular value decomposition writes an h x w image matrix as U * diag(s) * Vt, with singular values s in decreasing order. Keeping only the first k terms gives the best rank-k approximation of the image in the least-squares sense. Storing it needs k * (h + w + 1) numbers instead of h * w.

The test picture is a 48 by 48 image with a gradient background, a bright disc, a dark bar and a little noise. PSNR (peak signal-to-noise ratio, in decibels) summarizes the reconstruction error on a logarithmic scale; higher is better, and values above 30 to 40 dB usually look good.

Python Low-rank compression with SVD

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first singular value (about 6,900) dwarfs the others (1,255, 677, 474, ...), which means a rank-1 image, a plain product of a column pattern and a row pattern, already captures the overall brightness layout. Rank 4 stores 17 percent of the numbers and reaches 30.0 dB; rank 8 stores 34 percent and reaches 40.5 dB. At rank 24 the "compressed" version needs 2,328 numbers, which is more than the 2,304 pixels of the original: low-rank storage only pays off when k * (h + w + 1) is smaller than h * w, here for ranks up to 23.

This picture is unusually well suited to low-rank approximation because it is smooth with big flat shapes. Photographs with fine texture need much higher ranks, which is why practical codecs such as JPEG work on small blocks with a cosine transform and quantization instead. Still, the principle (represent a signal with the few most important basis patterns) is the same.

flowchart TB
    A["Compression idea"] --> B["Many images<br/>PCA basis shared by all"]
    A --> C["One image<br/>SVD of its own pixel matrix"]
    B --> D["Store weights per image<br/>+ one set of components"]
    C --> E["Store k columns, k rows<br/>and k singular values"]

Both approaches keep the strongest patterns and drop the rest; they differ in what is shared.

Using PCA safely

  • Fit on training data only. If you fit PCA on all images, including the test set, information leaks from the test set into the features. Fit it on the training split and apply transform to the test split (or put PCA in a scikit-learn Pipeline).
  • Scale consistently. All pixels in this lesson are on the same 0-to-1 scale, so no standardizing was needed. With mixed feature types, scale first.
  • PCA is linear. It finds straight directions. Strongly curved structure (a rotating object) needs more components than its true degrees of freedom.
  • Components are not parts. They have positive and negative regions, so they do not look like "eyes" or "stroke segments".
  • Variance is not the same as importance. A subtle feature that separates two classes may sit in a weak component, so check the effect on your actual task.

In the next chapter you will see networks that learn their own features instead of using fixed ones like HOG or PCA, starting with The Convolutional Layer.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A dataset has 1,000 images of 20 x 20 pixels. What is the shape of the matrix you give to PCA?

    PCA expects one row per sample and one column per feature. Each 20 x 20 image is flattened to 400 numbers, so the table is 1000 rows by 400 columns.
  2. 2. The cumulative explained variance reaches 0.95 at 29 components for the digits. What does this mean?

    Explained variance describes how much of the spread of the data lies along the kept components. It says nothing directly about accuracy, and the components are not individual pixels or images.
  3. 3. Why should PCA be fitted on the training set only when you also have a test set?

    The components would be shaped partly by the test images, so the test set would no longer be truly unseen. Fit on training data, then transform the test data with the fitted model.
  4. 4. A rank-k SVD approximation of a 100 x 100 image stores k * (100 + 100 + 1) numbers. For which k does it stop saving space?

    The original has 10,000 numbers; 201 * k reaches 10,000 at about k = 49.8. So from rank 50 the approximation stores more numbers than the image itself.
  5. 5. What do the images of the principal components (the eigen-images) show?

    Each component is a pattern that, when added to the mean image with a positive or negative weight, produces the main ways the images differ.

Summary

  • PCA works on a table: flatten each image to a row, subtract the mean image, and find orthogonal directions of decreasing variance. In scikit-learn: mean_, components_, explained_variance_ratio_, transform, inverse_transform.
  • Eigen-images (the components) are patterns of difference from the mean; any image is approximated by the mean plus a weighted sum of a few of them.
  • Choose k from the cumulative explained variance. For the digits, 21 components give 90 percent and 41 give 99 percent of the variance.
  • Reconstruction error falls as k rises and reaches zero at full rank. Discarding weak components also removes noise.
  • Eigenfaces apply this to aligned face images; the first components often capture lighting or head shape. The generated-faces example found 3 hidden factors in 256 pixels.
  • A single image can be compressed with a truncated SVD, but storage savings exist only below the break-even rank. Fit PCA on training data only.

Next: The Convolutional Layer.