Skip to content
Tutorials Center
My learning

Topics

Lesson 68 of 73 ยท Projects

Project: Anomaly Detection with Autoencoders

Detect anomalies with an autoencoder in NumPy: train on normal data only, score by reconstruction error, choose a threshold and compare with PCA and z-scores.

Some of the most valuable events are rare and, by definition, unlike anything you have seen much of: a fraudulent transaction, a failing bearing, a defective product on a conveyor belt, a network intrusion. You usually cannot train an ordinary classifier for them, because you have only a handful of examples (or none) and the next failure will look different from the last. Anomaly detection takes the opposite approach. Learn what normal looks like from the plentiful normal data, and flag anything that does not fit.

The autoencoder from Autoencoders is a natural tool. It is trained to compress its input through a narrow bottleneck and reconstruct it. Trained only on normal data, it becomes good at reconstructing normal inputs and poor at reconstructing anything else, so its reconstruction error is an anomaly score. In this project you build the whole pipeline on simulated sensor data, compare the autoencoder with two simple baselines, and then face the decisions that matter in practice: where to put the threshold, and what happens when the training data is not as clean as you hoped.

The idea in three steps

Train on normal data No anomaly labels are needed.
Reconstruct Squeeze through a bottleneck and rebuild.
Score Large reconstruction error means unusual.
1 / 2

The simulated plant

The data simulates a machine with 20 sensors. Behind them are just two hidden operating conditions (think speed and load). The 20 readings are smooth, nonlinear functions of those two numbers plus a little noise, so every normal reading lies near a two-dimensional curved sheet inside the 20-dimensional space. That is the situation an autoencoder is designed for: a high-dimensional input whose real information is in a few dimensions.

We define three kinds of anomaly, each deliberately different:

  • Spike. One sensor suddenly reads far too high or low. This is the easy case: a single value is out of range.
  • Broken correlation. Every sensor reading is individually normal, but ten of them come from a different operating condition than the other ten. The sensors no longer agree with each other. No individual value is out of range, so a threshold on any one sensor cannot see it.
  • Drift. All sensors shift by a small constant amount, as when a calibration slips.

The autoencoder is a dense network 20 -> 16 -> 2 -> 16 -> 20 with tanh units and a linear bottleneck of two numbers, trained with mean squared error and Adam, using only normal data. As baselines we use PCA with two components (the linear version of the same idea: project onto the best 2-D plane and measure what is left over) and the largest per-sensor z-score (how many standard deviations the most unusual sensor is from its mean).

To score a detector without choosing a threshold we use the area under the ROC curve (AUC): the probability that a randomly chosen anomaly gets a higher score than a randomly chosen normal reading. 1.0 is perfect and 0.5 is guessing.

Python Autoencoder versus PCA versus z-scores

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Training only on normal data, the autoencoder's reconstruction error on those data falls from 0.935 at the start to 0.012 after 400 epochs. Look at the AUC table. Spikes are found by everything (the z-score approach is weakest at 0.963). Drift is found perfectly by the autoencoder (1.000), by PCA nearly so (0.987), and less well by the z-scores (0.832), because a shift of a third of a standard deviation in every sensor is individually unremarkable. The broken correlation is the revealing case: the z-score method scores 0.585, barely better than guessing, because every single value is normal. PCA reaches 0.920 and the autoencoder 0.992. Models that understand how the sensors relate to each other can see what per-sensor checks cannot. The autoencoder beats PCA because the relationship between sensors here is curved, and a nonlinear bottleneck follows a curve while PCA fits a flat plane.

Choosing a threshold

An anomaly detector produces a score. Turning it into alarms requires a threshold, and the right threshold is a business decision, not a mathematical one. Never choose it using the anomalies in your test set, and in many real settings you have no labeled anomalies at all. The standard recipe is to score a held-out validation set of normal data and pick a high percentile of those scores: at the 99th percentile you accept that about 1% of normal readings will raise a false alarm.

The next program builds a realistic test set of 2,000 readings with about 5% anomalies of all three types, trains the autoencoder, and shows how the percentile moves the alarm count. Then it examines two training choices: how wide the bottleneck is, and what happens when the "normal" training set secretly includes anomalies.

Python Thresholds, bottleneck size and contaminated data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The test set has 102 anomalies among 2,002 readings, and the overall AUC is 0.988. The table shows the operating trade-off. At the 90th percentile of normal validation errors the detector catches 97.1% of the anomalies but raises 234 false alarms, so only 29.7% of its alarms are real. At the 99th percentile it catches 96.1% with 75 false alarms (56.6% precision). At the 99.9th percentile there are only 11 false alarms (89.7% precision) while still catching 94.1%. Because anomalies are rare, even a small false-alarm rate produces many false alarms in absolute numbers, which is the base-rate problem from Evaluation Metrics, and an operator who is flooded with false alarms stops reading them. A good threshold depends on what a missed anomaly costs against what an investigation costs.

The bottleneck experiment shows a different trade. With a code of only 1 number the autoencoder cannot represent the normal conditions (training error 0.145, AUC 0.913), and with 2 numbers, which match the true number of hidden conditions, the AUC is 0.988. In this small experiment wider codes (4, 8, 16) score even higher, 0.997 to 0.999, because the network is tiny and trained for only 300 epochs, so it cannot memorize its way to a perfect copy. With much bigger networks and long training an autoencoder can learn to reconstruct almost anything, including anomalies, and then the error stops being informative. That is why the bottleneck is the main safeguard, and why you should check anomaly detection on held-out anomalies when you have any, not only trust the training loss. Common ways to keep the model from learning a trivial copy are a narrow code, denoising training (corrupt the input and reconstruct the clean version), and regularization.

The last table covers contamination. If 5% of the "normal" training data are actually anomalies, the AUC with a bottleneck of 8 drops from 0.999 to 0.996, and with 20% of the training data contaminated by broken-correlation readings it drops to 0.980. The effect is gentle in this experiment but real: the model starts to treat what it has seen often as normal. Clean the training data where you can, for example by training, removing the highest-error samples and retraining.

The real-data recipe

For tabular data like transactions or sensor features, the same structure works with a few more layers. For images (a product surface, a medical scan), a convolutional autoencoder reconstructs the picture, and the per-pixel error image shows where the anomaly is. For time series, an LSTM or 1D-convolutional autoencoder reconstructs windows of the signal. This code is read-only:

Python
# Read-only: a dense autoencoder for tabular anomaly detection in PyTorch
import torch, torch.nn as nn

class AE(nn.Module):
    def __init__(self, n_features, code=4):
        super().__init__()
        self.encoder = nn.Sequential(nn.Linear(n_features, 32), nn.ReLU(), nn.Linear(32, code))
        self.decoder = nn.Sequential(nn.Linear(code, 32), nn.ReLU(), nn.Linear(32, n_features))
    def forward(self, x):
        return self.decoder(self.encoder(x))

model = AE(n_features=30)
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
for epoch in range(100):
    for (x,) in normal_loader:                                      # NORMAL data only, scaled with training statistics
        loss = nn.functional.mse_loss(model(x), x)
        opt.zero_grad(); loss.backward(); opt.step()

with torch.no_grad():
    score = ((model(x_new) - x_new) ** 2).mean(dim=1)               # reconstruction error per row
threshold = torch.quantile(validation_scores, 0.99)                 # 99th percentile of normal validation scores
alarms = score > threshold

Compare it with classical detectors before you commit: scikit-learn offers IsolationForest (isolates unusual points with random splits), OneClassSVM, and LocalOutlierFactor. They need little tuning and are often competitive on tabular data. Standard public data sets are the credit-card fraud data on Kaggle (284,807 transactions of which only 492 are fraud, about 0.17%), the MVTec AD collection of industrial images with defects, and the Numenta Anomaly Benchmark for time series.

Pitfalls checklist

  1. Scale the data with training statistics. A sensor in large units would dominate the reconstruction error otherwise.
  2. Evaluate with realistic prevalence. Report precision at your operating threshold on data with the true (tiny) fraction of anomalies, not on a 50/50 set.
  3. Beware drift in normal. Machines age and seasons change. A detector trained last year can raise alarms on perfectly healthy behavior today, so monitor and retrain.
  4. Per-sample error hides the cause. Look at the per-feature error (or the per-pixel error for images) to tell an operator which sensor or region looks wrong.
  5. Rare is not the same as bad. Unusual normal events (a holiday sales peak) can look like anomalies. Anomaly detection finds the unusual, and a human or another model decides if it matters.
  6. Supervised models when you can. If you have many labeled anomalies of the kinds you care about, an ordinary classifier usually beats unsupervised detection. Use anomaly detection for the unknown unknowns.

Ideas to extend the project

  • Add a fourth anomaly type, such as a sensor stuck at a constant value, and see which detectors catch it.
  • Train a denoising autoencoder by adding noise to the inputs, and compare the AUC.
  • Plot (or print) the per-sensor reconstruction error for a broken-correlation anomaly, and check that it highlights the sensors that changed.
  • Implement an isolation forest from scratch (random splits and path lengths) and add it to the table.
  • Retrain periodically on a sliding window of recent normal data and track how the threshold moves.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is an autoencoder trained only on normal data useful for anomaly detection?

    Having learned only the pattern of normal data, the model produces a large error on inputs that do not follow it.
  2. 2. Why did the per-sensor z-score method fail on the broken-correlation anomaly?

    A per-sensor check looks at values one at a time. The anomaly lay in the combination of values, which models of the joint structure (PCA, autoencoders) can detect.
  3. 3. How should the alarm threshold be chosen when there are no labeled anomalies?

    A validation set of normal data lets you fix the false-alarm rate, for instance the 99th percentile gives about 1% false alarms on normal readings.
  4. 4. At the 90th percentile threshold only 29.7% of alarms were real, although recall was 97.1%. Why?

    With about 5% anomalies and a 10% false-alarm rate on normal data, false alarms outnumber real detections.
  5. 5. What is the danger of anomalies hiding in the 'normal' training set?

    An autoencoder learns whatever it is shown often. Contamination makes some anomalies look normal, and AUC fell as it increased in the experiment.

Summary

  • Anomaly detection learns what normal looks like and flags what does not fit, so it needs no anomaly labels.
  • An autoencoder trained on normal data gives a reconstruction error that works as an anomaly score; its bottleneck is the main thing stopping it from copying everything.
  • It beat PCA and per-sensor z-scores most clearly on the broken-correlation anomaly (AUC 0.992 against 0.920 and 0.585), where each value is normal but the combination is not.
  • Choose the threshold from normal validation scores; because anomalies are rare, even a small false-alarm rate produces many false alarms, so measure precision at the true prevalence.
  • Contaminated training data weakens detection (AUC 0.999 to 0.980 at 20% contamination here); clean it and retrain as normal behavior drifts.
  • Compare with isolation forests and other classical detectors, and look at per-feature errors to explain each alarm.

Next project: Image Captioning.