Skip to content
Tutorials Center
My learning

Topics

Lesson 43 of 73 ยท Generative and Other Models

Autoencoders

Build an autoencoder in NumPy on handwritten digits: the bottleneck, reconstruction loss, a 2-D latent space plot, denoising and anomaly detection.

Every network so far learned a mapping from inputs to labels or numbers someone provided. An autoencoder needs no labels at all. It is trained to do something that sounds pointless: reproduce its own input. The trick that makes it useful is a narrow layer in the middle, the bottleneck, that forces the network to squeeze each input into a short summary (the code) and rebuild it from that summary alone.

Whatever the code manages to keep is, by construction, what matters most about the data. That makes autoencoders a tool for compression, for visualizing high-dimensional data, for cleaning noisy inputs and for spotting inputs that do not look like the rest. In this lesson you will build one in NumPy on 8x8 handwritten digits and try all four.

Squeeze, then rebuild

Encoder Compresses the input into a short code.
Bottleneck Too small to copy the input, so it must summarize.
Decoder Rebuilds the input from the code.

The training target is the input itself, so no labels are needed.

1 / 2

The architecture

An autoencoder has two halves. The encoder maps an input x (64 pixel values for a digit) down to a code z with only a few numbers. The decoder maps z back up to a reconstruction x_hat with the same shape as x. The loss compares the two, most simply with the mean squared error (see Loss Functions):

loss = mean((x_hat - x)^2)

flowchart LR
    X["Input x<br/>64 numbers"] --> E["Encoder"]
    E --> Z["Code z<br/>2 numbers"]
    Z --> D["Decoder"]
    D --> R["Reconstruction<br/>64 numbers"]
    R -. "compare with x" .-> X

The dotted line is the training signal: the output should match the input.

Without the bottleneck there would be nothing to learn: a network could pass the input straight through. The smaller the code, the more the network has to choose which features to keep. A bottleneck is not the only way to force this. You can also add noise, penalize large codes, or require sparse ones, but the narrow layer is the clearest idea.

A baseline without neural networks

Before using a neural network, it is worth knowing what a classic method achieves. PCA (principal component analysis) finds the best linear compression: it keeps the directions in which the data varies most. A linear autoencoder with squared error learns the same subspace, so PCA is the natural yardstick for our nonlinear one. The next example measures how well the top k PCA directions rebuild the digits, using the singular value decomposition (SVD). The dataset is scikit-learn's built-in set of 1,797 digit images, each 8x8 pixels.

Python The PCA reconstruction baseline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The error shrinks as the code grows: a code of 1 number leaves 0.0624 error per pixel, 2 numbers give 0.0524, 16 give 0.0110, and 64 numbers (the full image) give zero. Squeezing 64 pixels into 2 numbers loses a lot, so a good 2-number code is an achievement. These figures are the bar a nonlinear autoencoder has to beat.

The autoencoder in NumPy

The class below is a compact dense autoencoder: tanh in the hidden layers, a sigmoid at the output so reconstructions stay between 0 and 1 like the pixels, and mean squared error. fit runs mini-batch Adam, with backpropagation written out in a loop that works for any list of layer sizes (compare Backpropagation in NumPy). The only difference from a classifier is that the target is the input.

We use the layer sizes [64, 24, 2, 24, 64]: the code layer has two units, so every digit becomes a point on a plane.

Python A 2-D autoencoder and its latent space

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss falls from 0.189 to about 0.043, and the final error over all images is 0.0444. A 2-number code reconstructs the digits better than PCA's 2-number code (0.0524) because the network can use curved, nonlinear summaries. The left plot is the latent space: each point is one image, colored by its true digit, even though the network never saw a label. Digits of the same kind tend to fall near each other, while similar shapes overlap. The reconstructions on the right keep the rough shape but are blurry: two numbers cannot hold the fine detail.

Remember that the training never used the labels. The structure in the plot is something the network discovered because similar images need similar summaries.

Choosing the code size

A larger code gives better reconstructions, a smaller one gives stronger compression and a more meaningful summary. There is no free lunch, and the right size depends on the purpose:

Goal Typical code size
Visualization 2 or 3
Compression As small as reconstruction quality allows
Features for another model Large enough to keep what the next model needs
Anomaly detection Small enough that unusual inputs cannot be rebuilt

One more warning: an autoencoder with too much capacity (a code as big as the input and big hidden layers) can learn something close to the identity function and reconstruct everything, including noise and anomalies, perfectly. That is useless for the tasks below.

Denoising autoencoders

A denoising autoencoder changes one thing in training: it receives a corrupted input but is graded against the clean original. To succeed, it can no longer copy; it must learn what clean digits look like and project noisy inputs back onto that. The corruption can be Gaussian noise, randomly zeroed pixels (as in dropout, see Dropout and Data Augmentation) or masked patches.

The next example trains two networks of the same shape: one plain, one denoising, with fresh noise every few steps. Both then see noisy test images that neither was trained on, and we compare the reconstructions with the clean originals.

Python Plain versus denoising autoencoder

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The noisy test images are 0.0526 away from the clean ones. The plain autoencoder, which never saw noise, improves that only slightly (0.0418), because its mapping was learned on clean data. The denoising autoencoder gets down to 0.027. In the picture, the third row (plain) still shows speckles, while the fourth row (denoising) is smoother and closer to the top row. This idea, learning to undo a corruption, comes back in modern image generators; see VAEs, GANs and Diffusion Models.

Anomaly detection

An autoencoder trained only on "normal" data learns to reconstruct that data well. An input that is different uses patterns the network never learned, so it comes back with a large reconstruction error. Choosing a threshold on that error turns the autoencoder into an anomaly detector. This is used for finding defective products in images, unusual machine sensor readings or suspicious transactions, wherever anomalies are rare and hard to label in advance.

To simulate this, the next example trains only on digits 0 to 4 and then asks which unseen images look unusual. Digits 5 to 9 play the role of anomalies. We set the threshold so that 10% of normal held-out images are flagged, then see how many anomalies are caught.

Python Spotting digits the model has never seen

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The average error is 0.0319 for unseen normal digits and 0.0725 for the anomalies, more than twice as high. With the threshold at the 90th percentile of normal errors, about 10% of normal images are flagged (the false alarms we chose) and about 70% of the digits 5 to 9 are caught. The detector is imperfect, and the per-digit lines show why: digits 5 and 6 are caught almost every time (97% and 98%), but 8 and 9 only about 47% and 46% of the time, because they resemble normal digits (such as 3 and 4) closely enough that the network rebuilds them well, and a sharper threshold would trade fewer false alarms for more misses. Autoencoder anomaly detection always involves that trade-off, and the threshold should be tuned with examples of real anomalies when you have them.

Undercomplete, overcomplete, sparse and contractive

The shape of the code decides what an autoencoder can learn.

  • An undercomplete autoencoder has a code smaller than the input. It cannot copy the input, so it must keep only the most informative directions. This is the version built above, and it is the one to use for compression and dimensionality reduction. With linear layers it learns the same subspace as PCA; non-linear layers let it follow curved structure.
  • An overcomplete autoencoder has a code as large as or larger than the input. Left alone it can learn the identity function and nothing useful, so it needs a regularizer that stops it from simply copying.
  • A sparse autoencoder is overcomplete with a penalty (L1 on the code, or a KL penalty towards a small target activation) that keeps most code units near zero for any given input. Each input is then described by a few active units, which often become detectors of distinct features.
  • A contractive autoencoder adds the squared Frobenius norm of the encoder's Jacobian, ||โˆ‚h/โˆ‚x||ยฒ_F, to the loss. The code must change little when the input changes a little, so it learns representations that are robust to small noise, with sensitivity only along the directions where the data actually varies.
  • A denoising autoencoder (covered above) achieves a similar robustness by corrupting the input instead of penalizing the Jacobian.

The basic ("vanilla") autoencoder is simply the undercomplete one with no extra penalty. Reaching for an undercomplete design makes sense when you want compact codes; the regularized overcomplete designs make sense when you want interpretable or robust features.

Python Compare autoencoder types on 6-D data that lives on a plane

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table: the 2-unit undercomplete model reconstructs well because the data really has two dimensions, and the 1-unit one cannot. The overcomplete models reconstruct best, but only the sparse one turns many units off, and only the contractive one makes the code far less sensitive to small changes in the input.

Other autoencoder variants

  • Sparse autoencoders allow a wide code but penalize it so that only a few units are active at once.
  • Convolutional autoencoders use convolutions for the encoder and transposed convolutions for the decoder, which is the right choice for images (see Convolution, Pooling and Feature Maps).
  • Variational autoencoders make the code a probability distribution so that the decoder can generate new data; that is the topic of the next lesson.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the training target of a standard autoencoder?

    An autoencoder is trained to reproduce its input after passing it through a narrow code, so it needs no labels.
  2. 2. Why is the bottleneck important?

    If the code were as large as the input, the network could learn the identity function. A small code forces a useful summary.
  3. 3. What does a denoising autoencoder do differently during training?

    Grading the output against the clean version teaches the network what clean data looks like, so it learns to remove the corruption.
  4. 4. How does an autoencoder detect anomalies?

    Trained on normal data only, the network reconstructs unusual inputs poorly. A threshold on the reconstruction error then flags them.
  5. 5. A linear autoencoder trained with squared error learns essentially the same subspace as which classic method?

    The best linear k-dimensional compression under squared error is given by the top k principal components, so PCA is the natural baseline for autoencoders.

Summary

  • An autoencoder has an encoder, a narrow code (bottleneck) and a decoder, and is trained to reproduce its input, so it needs no labels.
  • The loss is usually the mean squared error between the reconstruction and the input.
  • A nonlinear autoencoder can beat PCA at the same code size; a 2-D code doubles as a map of the data.
  • Denoising autoencoders receive corrupted inputs and are graded against the clean originals.
  • Anomalies show up as large reconstruction errors, and the choice of threshold trades false alarms against misses.
  • Too much capacity lets an autoencoder copy everything, which makes it useless for these tasks.

Next: VAEs, GANs and Diffusion Models.