Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 40 ยท Advanced Topics

Autoencoders and GANs

Learn autoencoders for compression and denoising and GANs for generating data: encoder, bottleneck, generator, discriminator and the training loop, with NumPy demos.

Most of this course used supervised learning: examples with labels. Generative and representation-learning models work without labels, or learn to produce new data. Two classic families are autoencoders, which learn compact representations by reconstructing their own input, and GANs (generative adversarial networks), which learn to generate realistic samples through a contest between two networks. Both are good tests of whether you understand how loss functions and training loops fit together.

TensorFlow is not available in the browser runner. Keras code is read-only; runnable blocks use NumPy to demonstrate the mechanics, including a linear autoencoder trained by gradient descent.

Autoencoders

An autoencoder has two halves:

  • An encoder compresses the input x into a small code z (the bottleneck).
  • A decoder reconstructs an approximation x' from z.

It is trained with the input as its own target (loss = difference between x and x', typically MSE). Because the code is smaller than the input, the network cannot just copy; it must keep the most important structure.

flowchart LR
    X["input x<br/>784 numbers"] --> E["encoder<br/>Dense 128, Dense 32"]
    E --> Z["code z<br/>32 numbers"]
    Z --> D["decoder<br/>Dense 128, Dense 784"]
    D --> R["reconstruction x'"]
    X -.->|loss: x vs x'| R

No labels needed. The bottleneck forces compression.

Python
import keras
from keras import layers

encoder = keras.Sequential([
    keras.Input(shape=(784,)),
    layers.Dense(128, activation="relu"),
    layers.Dense(32, activation="relu"),
], name="encoder")

decoder = keras.Sequential([
    keras.Input(shape=(32,)),
    layers.Dense(128, activation="relu"),
    layers.Dense(784, activation="sigmoid"),
], name="decoder")

autoencoder = keras.Sequential([encoder, decoder])
autoencoder.compile(optimizer="adam", loss="mse")
autoencoder.fit(x_train, x_train, epochs=10, batch_size=256,
                validation_data=(x_test, x_test))     # input = target

A tiny autoencoder in NumPy

Data that truly lives in a low-dimensional space can be compressed with little loss. Here 10-dimensional points are generated from only 2 hidden factors, then a linear autoencoder with a 2-unit bottleneck learns to reconstruct them. (A linear autoencoder learns the same subspace as PCA.)

Python Training a linear autoencoder

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The reconstruction error falls close to the noise level because the 2-unit code captures both factors. If the true data had 5 factors, a 2-unit bottleneck could not reconstruct it well. The bottleneck size is the main knob.

Variants and uses

  • Denoising autoencoder: add noise to the input, but compute the loss against the clean original. It learns to remove noise.
  • Sparse autoencoder: add an L1 penalty on the code so few units are active.
  • Variational autoencoder (VAE): the encoder outputs a mean and variance and the code is sampled; an extra KL term keeps the code space smooth so you can generate new data by decoding random codes.
  • Uses: compression, denoising, anomaly detection (large reconstruction error means unusual input), pretraining features, and visualization.
Python Anomaly detection by reconstruction error

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

GANs: a contest between two networks

A GAN has:

  • A generator G that turns random noise z into fake samples.
  • A discriminator D that tells real samples from fakes (outputs a probability that its input is real).

They are trained together: D tries to classify correctly; G tries to make D believe its fakes are real. At equilibrium the fakes look like the real distribution and D guesses at 50 percent.

flowchart LR
    N["random noise z"] --> G["Generator"]
    G --> F["fake sample"]
    R["real sample"] --> D["Discriminator"]
    F --> D
    D --> P["real or fake?"]
    P -->|train D to be right| D
    P -->|train G to fool D| G

Two optimizers, two losses, alternating updates.

One training step alternates:

  1. Train D on a batch of real samples (label 1) and fakes (label 0).
  2. Train G so that D(G(z)) moves toward 1, keeping D's weights fixed.
Python
class GAN(keras.Model):
    def __init__(self, generator, discriminator, latent_dim):
        super().__init__()
        self.generator, self.discriminator = generator, discriminator
        self.latent_dim = latent_dim

    def compile(self, g_opt, d_opt, loss_fn):
        super().compile()
        self.g_opt, self.d_opt, self.loss_fn = g_opt, d_opt, loss_fn

    def train_step(self, real):
        batch = tf.shape(real)[0]
        z = tf.random.normal((batch, self.latent_dim))
        fake = self.generator(z)

        with tf.GradientTape() as tape:                       # discriminator step
            real_out = self.discriminator(real, training=True)
            fake_out = self.discriminator(fake, training=True)
            d_loss = (self.loss_fn(tf.ones_like(real_out), real_out)
                      + self.loss_fn(tf.zeros_like(fake_out), fake_out))
        grads = tape.gradient(d_loss, self.discriminator.trainable_variables)
        self.d_opt.apply_gradients(zip(grads, self.discriminator.trainable_variables))

        z = tf.random.normal((batch, self.latent_dim))
        with tf.GradientTape() as tape:                       # generator step
            out = self.discriminator(self.generator(z), training=True)
            g_loss = self.loss_fn(tf.ones_like(out), out)     # "pretend fakes are real"
        grads = tape.gradient(g_loss, self.generator.trainable_variables)
        self.g_opt.apply_gradients(zip(grads, self.generator.trainable_variables))
        return {"d_loss": d_loss, "g_loss": g_loss}

This uses the custom train_step pattern. Note the generator trains on the label 1 for fakes: it wants the discriminator to say "real".

A one-dimensional GAN in NumPy

To see the dynamic, make real data a bell curve with mean 4. The generator is G(z) = a*z + b and the discriminator is a logistic regression on one number. The generator should learn b โ‰ˆ 4.

Python A tiny GAN learning a mean

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The generator's mean overshoots to about 4.75, swings back and settles near the real mean of 4. Its scale |a| shrinks to about 0.08 instead of 0.5: the generator produces nearly the same value every time, a miniature mode collapse. This is a small taste of why GANs are hard: training is a moving target, and D and G can oscillate or collapse.

Why GANs are hard to train

  • Instability: two networks chase each other; losses do not decrease smoothly and are poor progress indicators.
  • Mode collapse: the generator learns to produce only a few kinds of samples that fool D.
  • Balance: if D becomes perfect, G gets no useful gradient.

Common fixes are label smoothing, different learning rates, spectral normalization, Wasserstein losses and careful architecture choices (DCGAN uses strided convolutions and batch normalization). Today diffusion models often replace GANs for image quality, but GAN ideas remain widely used.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the target (y) when training an autoencoder with fit()?

    An autoencoder reconstructs its input, so x is used as both input and target.
  2. 2. How can an autoencoder detect anomalies?

    The model learns the structure of normal data, so abnormal inputs are reconstructed badly.
  3. 3. In GAN training, which label does the generator step use for fake samples?

    The generator is rewarded when the discriminator outputs 'real' for its fakes.
  4. 4. What is mode collapse?

    The generator finds a few samples that fool the discriminator and stops covering the whole data distribution.

Summary

  • An autoencoder compresses input through a bottleneck and reconstructs it; the loss compares output with input.
  • Variants include denoising, sparse and variational autoencoders; uses include compression and anomaly detection.
  • A GAN pits a generator against a discriminator; they are trained in alternating steps with their own optimizers.
  • GAN training is unstable (oscillation, mode collapse), and loss curves alone do not show quality.
  • Both models are natural uses of custom train_step.

Next: Project: Image Classifier