Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 45 ยท Generative Models

Autoencoders and Variational Autoencoders

Build autoencoders in PyTorch: encoder, bottleneck and decoder, reconstruction loss, denoising, sparse and convolutional variants, and the variational autoencoder.

An autoencoder is a network trained to copy its input to its output, through a narrow middle. That sounds pointless until you notice the catch: because the middle layer, the bottleneck, is smaller than the input, the network cannot just pass the data along. It must learn a compact summary from which the input can be rebuilt. The summary, called the latent code or embedding, is useful for compression, denoising, anomaly detection and, in its variational form, for generating new data.

Autoencoders are trained without labels (the input is its own target), which makes them an unsupervised method. PyTorch cannot run in the browser, so the PyTorch code is read-only; the live examples use NumPy to run the same ideas, including a linear autoencoder that you can compare with PCA.

The structure

An autoencoder has two halves.

  • The encoder maps the input x to a code z with fewer dimensions.
  • The decoder maps z back to a reconstruction x_hat with the shape of x.

Training minimizes the reconstruction loss, usually nn.MSELoss for real-valued data (or BCEWithLogitsLoss for pixels in [0, 1]). The loss compares x_hat with x; no labels appear anywhere.

flowchart LR
    X["input x<br/>784"] --> E["encoder"]
    E --> Z["latent z<br/>32"]
    Z --> D["decoder"]
    D --> R["reconstruction<br/>784"]
    R -.->|"loss: compare with x"| X

The bottleneck forces the network to keep only what matters.

A fully connected autoencoder in PyTorch

Python
import torch
import torch.nn as nn

class Autoencoder(nn.Module):
    def __init__(self, in_dim=784, hidden=128, latent=32):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(in_dim, hidden), nn.ReLU(),
            nn.Linear(hidden, latent),
        )
        self.decoder = nn.Sequential(
            nn.Linear(latent, hidden), nn.ReLU(),
            nn.Linear(hidden, in_dim), nn.Sigmoid(),   # pixels in [0, 1]
        )

    def forward(self, x):
        return self.decoder(self.encoder(x))

model = Autoencoder()
loss_fn = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

for images, _ in loader:                 # the labels are ignored
    x = images.view(images.size(0), -1)  # flatten 28x28 to 784
    loss = loss_fn(model(x), x)          # the target is the input itself
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

The loop is the standard one from The Training Loop; only the target changes. To look at the results, show a few originals next to model(x): sharp copies mean the code kept enough information.

Autoencoders and PCA

If the encoder and decoder are linear (no activations), the best possible autoencoder spans the same subspace as principal component analysis (PCA). Non-linear autoencoders generalize PCA: they can follow curved structure that a flat subspace cannot. The next editor trains a linear autoencoder by gradient descent and compares it with the PCA optimum computed by SVD.

Python A linear autoencoder rediscovers PCA

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two subspaces coincide, so the "learned features" of a linear autoencoder are exactly PCA's directions.

Types of autoencoders

Type What changes Why
Undercomplete (basic) Bottleneck smaller than input Forces compression
Denoising Feed a corrupted input, reconstruct the clean one Learns robust features instead of copying
Sparse Penalize active code units (an L1 term or KL to a target activation) Few units fire per input, giving interpretable features; the bottleneck can even be wider than the input
Convolutional Conv2d encoder, ConvTranspose2d decoder Keeps spatial structure of images with far fewer weights
Contractive Penalize the encoder's sensitivity to small input changes Smooth, stable codes
Variational (VAE) Code is a probability distribution Can generate new data

A denoising autoencoder only changes the training step:

Python
noisy = x + 0.3 * torch.randn_like(x)            # corrupt the input
loss = loss_fn(model(noisy.clamp(0, 1)), x)      # compare with the CLEAN input

A convolutional autoencoder halves the image with strided convolutions and doubles it back with transposed convolutions:

Python
encoder = nn.Sequential(
    nn.Conv2d(1, 16, 3, stride=2, padding=1), nn.ReLU(),    # 28 -> 14
    nn.Conv2d(16, 32, 3, stride=2, padding=1), nn.ReLU(),   # 14 -> 7
)
decoder = nn.Sequential(
    nn.ConvTranspose2d(32, 16, 3, stride=2, padding=1, output_padding=1), nn.ReLU(),  # 7 -> 14
    nn.ConvTranspose2d(16, 1, 3, stride=2, padding=1, output_padding=1), nn.Sigmoid(),  # 14 -> 28
)

A sparse autoencoder adds a penalty to the loss, for example loss = mse + 1e-3 * z.abs().mean(), so most latent units stay near zero.

What the bottleneck buys you

Two practical uses follow directly from "reconstruct only what the code can express":

  • Denoising. Noise is spread over all directions of the input while the signal lives on a low-dimensional surface, so a bottleneck keeps the signal and drops most of the noise.
  • Anomaly detection. An autoencoder trained on normal data reconstructs normal inputs well and strange inputs badly. A high reconstruction error flags an anomaly: fraud, a defective part, a failing machine.
Python Denoising and anomaly detection with a bottleneck

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Variational autoencoders

A plain autoencoder's latent space has holes: a random code usually decodes to garbage, because nothing made the codes cover the space smoothly. A variational autoencoder (VAE) fixes this by making the encoder output a distribution instead of a point.

  1. The encoder outputs two vectors: a mean mu and a log-variance logvar.
  2. A code is sampled: z = mu + sigma * eps with eps ~ N(0, 1) and sigma = exp(0.5 * logvar). This is the reparameterization trick: the randomness sits in eps, so gradients can flow through mu and sigma during backpropagation.
  3. The decoder reconstructs x from z.
  4. The loss has two terms: reconstruction loss plus the KL divergence between the encoder's distribution N(mu, sigma^2) and a standard normal prior N(0, 1). The KL term pulls all codes toward one smooth cloud.
Python
class VAE(nn.Module):
    def __init__(self, in_dim=784, hidden=256, latent=16):
        super().__init__()
        self.enc = nn.Sequential(nn.Linear(in_dim, hidden), nn.ReLU())
        self.mu = nn.Linear(hidden, latent)
        self.logvar = nn.Linear(hidden, latent)
        self.dec = nn.Sequential(nn.Linear(latent, hidden), nn.ReLU(),
                                 nn.Linear(hidden, in_dim))        # logits

    def forward(self, x):
        h = self.enc(x)
        mu, logvar = self.mu(h), self.logvar(h)
        z = mu + torch.exp(0.5 * logvar) * torch.randn_like(mu)    # reparameterization
        return self.dec(z), mu, logvar

def vae_loss(logits, x, mu, logvar):
    recon = nn.functional.binary_cross_entropy_with_logits(logits, x, reduction="sum")
    kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())   # closed form
    return recon + kl

To generate new data after training, skip the encoder: sample z = torch.randn(16, latent) and pass it through the decoder. Because training pushed the codes toward N(0, 1), such samples decode to plausible outputs. The closed-form KL above is exact; the editor checks it against a random-sampling estimate.

Python The VAE KL term and the reparameterization trick

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Autoencoder versus VAE versus PCA

PCA Autoencoder VAE
Mapping Linear Non-linear Non-linear, probabilistic
Latent space Flat subspace Unconstrained Smooth, close to N(0, 1)
Generates new data Not really Poorly Yes
Fit Closed form (SVD) Gradient descent Gradient descent with a KL term

Sequence-to-sequence models

The encoder-decoder idea is not limited to images. A sequence-to-sequence (seq2seq) model uses an encoder (an RNN or transformer) to compress a whole input sequence into a context, and a decoder to generate an output sequence of a different length from it: translation, summarization, speech recognition. See Recurrent Networks and Transformers for the building blocks.

Warning: A bottleneck that is too wide lets the network learn the identity function and nothing useful. Too narrow and reconstructions are blurry. Check the reconstruction error and try a few sizes.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the training target of an autoencoder?

    The loss compares the reconstruction with the original input, so no labels are needed.
  2. 2. What is the main purpose of a denoising autoencoder's training trick?

    Reconstructing clean data from corrupted data forces the network to learn structure rather than copy.
  3. 3. Why does a VAE use the reparameterization trick?

    Writing z = mu + sigma * eps moves the randomness into eps, so z is a differentiable function of mu and sigma.
  4. 4. How do you generate new samples from a trained VAE?

    The KL term makes the codes resemble a standard normal, so random draws decode to plausible outputs.
  5. 5. Why can reconstruction error detect anomalies?

    The bottleneck only keeps patterns seen in training, so off-pattern inputs come back with a large error.

Summary

  • An autoencoder is an encoder, a bottleneck and a decoder trained to reconstruct its input with a loss such as MSE.
  • A linear autoencoder spans the same subspace as PCA; non-linear ones generalize PCA.
  • Variants: denoising, sparse, convolutional and contractive autoencoders.
  • Bottlenecks give denoising and, via reconstruction error, anomaly detection.
  • A VAE outputs mu and logvar, samples with the reparameterization trick and adds a KL term, so the decoder can generate new data.

Next: pit two networks against each other in Generative Adversarial Networks.