Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 45 ยท Generative Models

Generative Adversarial Networks (GANs)

Understand GANs and build one in PyTorch: generator and discriminator, the minimax loss, a training loop, common failures, and variants such as cGAN, WGAN, CycleGAN and StyleGAN.

A generative adversarial network (GAN) learns to create data by playing a game. One network, the generator, turns random noise into fake samples. A second network, the discriminator, looks at samples and guesses whether each is real or fake. The generator tries to fool the discriminator; the discriminator tries not to be fooled. As both improve, the generator's fakes become so good that the discriminator can only guess.

GANs produced the first photorealistic generated faces and remain the clearest example of two networks training against each other. This lesson shows the idea, the loss, a complete PyTorch training loop (read-only, since PyTorch cannot run in the browser), and a tiny GAN in NumPy that you can run and watch learn. It builds on Autoencoders and The Training Loop.

The two players

  • Generator G: takes a noise vector z (for example 100 numbers drawn from a standard normal) and outputs a sample G(z), such as a 28 by 28 image. It never sees real data directly; it learns only through the discriminator's feedback.
  • Discriminator D: takes a sample and outputs the probability that it is real. It is an ordinary classifier trained on real samples (label 1) and generated samples (label 0).
flowchart LR
    Z["noise z"] --> G["Generator"]
    G --> F["fake sample"]
    R["real data"] --> D["Discriminator"]
    F --> D
    D --> P["real or fake?"]
    P -.->|"feedback"| G

Both networks are trained from the same verdict: D to be right, G to make D wrong.

The minimax game

The two goals combine into one value function:

Text
min_G max_D  E[ log D(x) ] + E[ log(1 - D(G(z))) ]

D wants to maximize it: output near 1 on real x and near 0 on fakes. G wants to minimize it: make D(G(z)) large. This is a minimax (zero-sum) game. In theory, for a fixed G the best discriminator is D*(x) = p_real(x) / (p_real(x) + p_fake(x)), and at the equilibrium the fake distribution equals the real one, so D* is 0.5 everywhere and cannot tell them apart.

In practice the generator uses the non-saturating loss: instead of minimizing log(1 - D(G(z))), which has tiny gradients when D is confident the sample is fake (exactly the situation early in training), it maximizes log D(G(z)). In code that is just binary cross-entropy against the label "real".

Python The best discriminator and the equilibrium

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When the fakes are far from the real data, D* separates them sharply (values near 0 and 1), leaving the generator an easy signal. When they match, D* is 0.5 and the discriminator loss settles at 2 ln 2, about 1.386.

A GAN in PyTorch

Here is a complete, minimal GAN for 28 by 28 images (read-only):

Python
import torch
import torch.nn as nn

latent_dim = 100

G = nn.Sequential(
    nn.Linear(latent_dim, 256), nn.LeakyReLU(0.2),
    nn.Linear(256, 512), nn.LeakyReLU(0.2),
    nn.Linear(512, 784), nn.Tanh(),            # pixels scaled to [-1, 1]
)
D = nn.Sequential(
    nn.Linear(784, 512), nn.LeakyReLU(0.2),
    nn.Linear(512, 256), nn.LeakyReLU(0.2),
    nn.Linear(256, 1),                         # a logit: real vs fake
)

bce = nn.BCEWithLogitsLoss()
opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
opt_d = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))

for real, _ in loader:
    n = real.size(0)
    real = real.view(n, -1)                    # normalized to [-1, 1] to match Tanh
    ones, zeros = torch.ones(n, 1), torch.zeros(n, 1)

    # 1) Train the discriminator
    z = torch.randn(n, latent_dim)
    fake = G(z).detach()                       # detach: do not update G here
    loss_d = bce(D(real), ones) + bce(D(fake), zeros)
    opt_d.zero_grad(); loss_d.backward(); opt_d.step()

    # 2) Train the generator: make D call the fakes real
    z = torch.randn(n, latent_dim)
    loss_g = bce(D(G(z)), ones)                # non-saturating loss
    opt_g.zero_grad(); loss_g.backward(); opt_g.step()

Details that matter:

  • detach() during the discriminator step stops gradients from flowing into the generator; otherwise one backward pass would update both networks incorrectly.
  • Tanh on the generator output means real images must be scaled to [-1, 1], not [0, 1].
  • LeakyReLU in the discriminator avoids dead units; Adam with beta1 = 0.5 is the common recipe for stable GAN training.
  • Convolutional GANs (DCGAN) replace the dense layers: the generator uses ConvTranspose2d with batch normalization to grow from a 1x1 noise map to an image, and the discriminator uses strided Conv2d layers. The generator and discriminator should stay roughly balanced in strength.
  • Save generated samples from a fixed batch of noise every epoch to watch progress.

Watching a GAN learn

The next editor trains a miniature GAN entirely in NumPy. The "real" data is a one-dimensional Gaussian with mean 4 and standard deviation 1.25. The generator is G(z) = a*z + b (two parameters), and the discriminator is a logistic function of [x, x^2, 1]. The gradients are written by hand, and the loop mirrors the PyTorch one: a discriminator step, then a generator step with the non-saturating loss.

Python A tiny GAN learns a Gaussian

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The generator starts at mean 0 and std 1 and moves toward the real statistics. It hovers around them rather than settling exactly, which is typical of GANs: two players chase each other, so the losses do not simply fall the way they do in ordinary training. Averaging the generator's weights over time, as the last column does, gives a steadier result.

Training problems

GANs are famously hard to train. Watch for:

  • Mode collapse. The generator finds a few outputs that fool D and produces only those; diversity vanishes.
  • A discriminator that wins too early. If D is perfect, G gets almost no gradient and stops learning. Balance the two: lower D's learning rate, add noise to its inputs, or use label smoothing (real labels 0.9 instead of 1).
  • Oscillation and instability. Losses can swing without converging; the loss value alone does not tell you the quality of samples, so inspect images.
  • Evaluation is hard. Metrics such as the Frechet Inception Distance (FID) compare feature statistics of real and generated sets.

GAN variants

Variant Idea
Vanilla GAN The original game with dense layers and the loss above
DCGAN Convolutional generator and discriminator with batch norm; the stable baseline recipe
Conditional GAN (cGAN) Both networks also receive a label y (class, text, a sketch), so you can ask for "a 7"
Wasserstein GAN (WGAN) The discriminator becomes a "critic" scoring realness with no sigmoid; the loss approximates the Wasserstein distance, and weight clipping or a gradient penalty (WGAN-GP) keeps it well behaved. Gives smoother gradients and a loss that tracks quality
LAPGAN A Laplacian pyramid of GANs: each level adds the finer detail of a bigger image to an upsampled coarse one
Progressive GAN Starts at low resolution and grows both networks layer by layer to high resolution
CycleGAN Translates images between two domains without paired examples (horses to zebras) using two generators and a cycle-consistency loss: translating there and back should return the original
StyleGAN A style-based generator that maps noise to a style vector and injects it at every layer, separating coarse traits (pose) from fine ones (hair texture) and producing high-resolution faces

In a conditional GAN, the label is typically one-hot encoded (or embedded with nn.Embedding) and concatenated to the noise for the generator and to the sample for the discriminator:

Python
labels_onehot = nn.functional.one_hot(labels, num_classes=10).float()
fake = G(torch.cat([z, labels_onehot], dim=1))          # generator input: noise + label
score = D(torch.cat([fake, labels_onehot], dim=1))      # discriminator also sees the label

The WGAN critic loss replaces the cross-entropy:

Python
loss_critic = -(critic(real).mean() - critic(fake).mean())
loss_gen = -critic(G(z)).mean()

Advantages and limitations

GANs generate sharp, realistic samples, need no explicit probability model, and the generator is fast at sampling (one forward pass). They are unstable to train, can drop modes, and give no direct likelihood. Since 2021, diffusion models have overtaken GANs for many image-generation tasks, but the adversarial idea lives on in super-resolution, style transfer and as a training signal inside other systems.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In a GAN, what does the discriminator output?

    The discriminator is a binary classifier separating real samples from generated ones.
  2. 2. Why is fake = G(z).detach() used in the discriminator step?

    The discriminator update should only change D. Detaching cuts the graph back to G.
  3. 3. What is mode collapse?

    The generator keeps producing the few outputs that fool the discriminator, losing diversity.
  4. 4. What extra input does a conditional GAN give both networks?

    The label lets you control what is generated, for example the digit shown.
  5. 5. What do CycleGAN's two generators and cycle-consistency loss allow?

    Translating A to B and back should recover the original, which lets the model learn from unpaired sets of images.

Summary

  • A GAN pits a generator (noise to samples) against a discriminator (real or fake) in a minimax game.
  • Train D on real and detached fake batches, then train G to make D say "real" with the non-saturating loss.
  • At the theoretical equilibrium the fake distribution equals the real one and D = 0.5 everywhere.
  • Typical problems are mode collapse, an overpowering discriminator and unstable losses.
  • Variants: DCGAN, conditional GAN, WGAN, LAPGAN, Progressive GAN, CycleGAN and StyleGAN.

Next: adapt big pretrained models cheaply in Fine-Tuning and LoRA.