Skip to content
Tutorials Center
My learning

Topics

Lesson 44 of 73 ยท Generative and Other Models

VAEs, GANs and Diffusion Models

Learn how generative models create new data: the VAE reparameterization trick and KL term, the GAN game, and the diffusion forward and reverse process.

Most of the models in this course are discriminative: they look at data and answer a question about it. A generative model learns what the data itself looks like, well enough to produce new examples that were never in the training set: new faces, new sentences, new molecules. Behind nearly every image generator you have heard of is one of three ideas, or a blend of them.

This lesson gives you each idea at the smallest scale where it can still run and teach something: a variational autoencoder on one-dimensional data, a GAN that learns a bell curve, and a diffusion process on a tiny image. You will see the mechanics (the reparameterization trick, the KL term, the generator-discriminator game, the noising schedule) rather than the huge networks that make them famous.

Three ways to make new data

VAE Encode to a distribution, then decode samples from it.
GAN A forger and a detective train each other.
Diffusion Learn to remove noise, then start from pure noise.

All three learn a distribution you can draw samples from.

1 / 2

Variational autoencoders

In the previous lesson every input was squeezed to a single point in the code space. That is a problem for generation: if you pick a random point in the code space and decode it, you usually land in a gap that the network never trained on, and the output is garbage.

A variational autoencoder (VAE) fixes this by changing what the encoder outputs. Instead of one code z, the encoder produces the parameters of a distribution: a mean mu and a log-variance logvar. The code is then sampled from that Gaussian, and the decoder rebuilds the input from the sample. Because nearby codes must all decode to similar things, the code space fills up smoothly. A second term in the loss pulls every distribution toward the standard normal N(0, 1), so that after training you can draw z from N(0, 1), decode it, and get a plausible new example.

The reparameterization trick

There is one obstacle. Sampling z from N(mu, sigma^2) is a random operation, and backpropagation cannot pass a gradient through a random draw. The fix is to move the randomness out of the way:

z = mu + sigma * eps, with eps ~ N(0, 1)

Now eps is an input, like the data, and z is an ordinary differentiable function of mu and sigma. Gradients flow through mu and sigma back into the encoder. The first half of the next example checks this numerically: to find the gradient of E[(z - 3)^2] with respect to mu and sigma, differentiate through z = mu + sigma * eps and average over samples.

The KL term

The loss of a VAE has two parts:

loss = reconstruction error + KL(q(z|x) || N(0, 1))

The KL divergence measures how far the encoder's distribution is from the standard normal. For a one-dimensional Gaussian it has a closed form:

KL = -0.5 * (1 + logvar - mu^2 - exp(logvar))

It is zero only when mu = 0 and sigma = 1. The two terms pull in opposite directions: reconstruction wants each input to have its own precise, narrow code, and KL wants all codes to look the same. The balance produces a smooth, usable latent space. (The full objective is called the evidence lower bound, or ELBO; the loss above is its negative.) Weighting the KL term with a factor beta shifts the balance, and a larger beta gives a smoother space at the price of blurrier reconstructions.

Python Reparameterization and the KL term

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The sampled gradients match the exact ones: -4.0 for mu and about 1.0 for sigma, so the trick works. The KL table shows the closed form against a Monte Carlo estimate; they agree to about two decimals. A code that is exactly N(0, 1) costs 0, shifting the mean to 1 costs 0.5, and a tight, off-center distribution (mu = 2, sigma = 0.5) costs 2.318 nats.

Training a tiny VAE

The next example trains a real VAE, with the backward pass written by hand, on one-dimensional data: numbers clustered around -2 and +2. The latent space has a single dimension. After training we throw away the encoder, draw z from N(0, 1), and decode. If the VAE has learned the data, the generated numbers should also land in two clusters of roughly equal size.

Python A VAE that generates from pure noise

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The reconstruction error falls from 4.272 to about 0.23, while the KL term settles near 0.95 nats: the code carries about one nat of information (enough to say "left cluster or right cluster" and a little more) and no more. Decoding draws from the prior gives 50% of the values below zero and 50% above, matching the data's two clusters. The generated values have a standard deviation of 1.79 against 2.03 in the data: the samples sit around the right places, but a few land in between, and the clusters are slightly narrower. On images, this smoothing is what makes plain VAEs blurry.

Generative adversarial networks

A GAN frames generation as a game between two networks.

  • The generator takes random noise and produces a fake sample.
  • The discriminator looks at a sample and outputs the probability that it is real.

The discriminator is trained as a binary classifier: real examples labeled 1, generated ones labeled 0. The generator is trained to make the discriminator say "real" about its fakes, with gradients flowing through the discriminator into the generator. Each improvement in one forces the other to improve. At the ideal equilibrium the generator's distribution equals the data's, and the discriminator can only guess (output 0.5 everywhere).

In practice, generators use the non-saturating loss -log D(G(z)) instead of log(1 - D(G(z))), because it gives strong gradients early in training when the discriminator easily wins.

The example below is the whole game in one dimension: real data is a bell curve around 4 with standard deviation 0.8. The generator is x = w * z + b (the simplest possible generator), and the discriminator is a small network. Watch the generator's mean and spread move toward the target.

Python A GAN learning a bell curve

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The generator starts at mean 1 and spread 1, moves its mean up toward 4, and finishes close to the target (mean 3.99, std 0.77 against 4.00 and 0.80). The discriminator's scores for real and fake data converge to about 0.5, the "I cannot tell" equilibrium. Look at step 600, though: the generator's spread dropped to 0.05 for a while before recovering. That is a miniature mode collapse: the generator briefly found a single value that fools the current discriminator and stopped covering the whole distribution. GANs are known for this and other problems:

  • Instability. The two networks chase each other, so the loss is not a steady decrease and does not tell you when to stop.
  • Mode collapse. The generator produces only a few kinds of samples.
  • Balance. If the discriminator becomes too strong too early, the generator gets no useful gradient.

Better losses, normalization, and careful architectures made GANs work well for images, and they are still used where one fast forward pass per sample matters.

The minimax loss and the vanilla GAN

The original ("vanilla") GAN is a two-player minimax game over one value function:

TEXT
min_G max_D  E_x[ log D(x) ]  +  E_z[ log(1 - D(G(z))) ]

The discriminator D tries to make this large: output near 1 on real x, near 0 on fakes. The generator G tries to make it small by making D(G(z)) large. At the theoretical optimum the generator's distribution equals the data distribution and D outputs 0.5 everywhere. Early in training, when D easily spots fakes, log(1 โˆ’ D(G(z))) is flat and gives G almost no gradient, so in practice the generator minimizes the non-saturating loss โˆ’log D(G(z)) instead. The vanilla GAN uses fully connected layers; the variants below are the same game with better architectures. Training progression looks like this: at first the fakes are noise and D wins, then the fakes gain coarse structure and D's loss rises, and training stalls near an equilibrium where D is right about half the time.

GAN variants: DCGAN, cGAN, CycleGAN, StyleGAN### GAN variants: DCGAN, cGAN, CycleGAN, StyleGAN

The 1-D game above is the skeleton. Almost every GAN you will read about is that game with a better body, an extra input, or a repaired loss.

  • DCGAN (deep convolutional GAN) was the first recipe that reliably trained on images. The generator turns the noise vector into a small feature map (for example 4 by 4 with 512 channels) and then applies transposed convolutions with stride 2, each doubling the height and width while halving the channels. The discriminator mirrors it with strided convolutions. The recipe: no pooling, batch normalization in both networks (except the generator's output and the discriminator's input), ReLU in the generator with a tanh output, and LeakyReLU in the discriminator.
  • Conditional GAN (cGAN) adds a label y (a class, a text embedding, or even another image) to both networks. The generator sees [z, y] and the discriminator sees (x, y), so it learns to reject a realistic picture with the wrong label. Now you can ask for "a 7" rather than "a digit". Pix2Pix is a cGAN whose condition is an image, such as a sketch or a segmentation map, and it needs paired examples.
  • CycleGAN translates between two image domains (horses and zebras, summer and winter) without paired examples. It trains two generators, G: A -> B and F: B -> A, and adds a cycle-consistency loss: translating to the other domain and back must return the original, F(G(a)) โ‰ˆ a. Without that constraint G could output any plausible zebra and ignore the horse it was given.
  • StyleGAN changes the generator. Noise is first mapped by a small network into an intermediate "style" vector w, which is injected into every layer to control features at that scale (coarse layers set pose and face shape, fine layers set hair and skin texture), plus extra per-layer noise for stochastic details. The result is high-resolution, controllable faces, and moving w between two images blends their styles.
Python DCGAN shapes and cGAN inputs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each transposed convolution ((in - 1) * stride - 2 * pad + kernel) takes the map from 4 to 8 to 16 to 32 to 64 pixels, so a handful of numbers describes the whole generator. The weights concentrate in the early, narrow-and-deep layers (2.1 million of the 3.6 million sit in the first transposed convolution), and the last layer that makes the RGB picture needs only 3,072. In the cGAN, two rows share one label but have different noise: the label chooses what to draw, the noise chooses which one.

  • LAPGAN (Laplacian pyramid GAN) generates a picture coarse to fine. Build a Laplacian pyramid of images (a tiny blurry version, then the detail that must be added at each larger scale). One GAN produces the smallest image; a series of conditional GANs, each seeing the upsampled result so far, adds the missing high-frequency residual at the next scale. Splitting the job into several small, easy ones was an early answer to unstable large-image training, before progressive growing and StyleGAN.

Why GANs are hard to train, and the fixes### Why GANs are hard to train, and the fixes

Three problems keep coming back:

  • Vanishing generator gradients. If the discriminator becomes too good, it separates real from fake perfectly and the loss tells the generator nothing about how to improve. Training the original objective is tied to the Jensen-Shannon divergence, which is flat when the two distributions do not overlap. WGAN replaces it with the Wasserstein ("earth mover's") distance, which keeps shrinking as the generator gets closer. The critic must be kept smooth (1-Lipschitz), by weight clipping in the original paper, a gradient penalty (WGAN-GP) or spectral normalization.
  • Mode collapse. The generator finds one output (or a few) that fools the discriminator and keeps producing it. Every sample looks real, but the variety is gone. Remedies include WGAN-style losses, minibatch discrimination (let the discriminator see statistics of a whole batch), unrolled updates and diversity regularizers.
  • No single loss to watch. The two losses move against each other, so a flat curve does not mean the model is bad. You judge by looking at samples and by metrics such as FID (Frechet inception distance), which compares feature statistics of real and generated images, and by precision and recall measures that separate quality from coverage.
Python JS vs Wasserstein, a collapsed generator, and the cycle loss

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first table is the reason WGAN exists: the Jensen-Shannon divergence reads 0.693 whether the fake samples are 1 or 10 units away, so it cannot say which way to move, while the Wasserstein distance equals the distance itself. The second part shows why mode collapse is sneaky. The collapsed generator's samples all look perfectly real (100%), yet it covers one of the four modes, so a per-sample quality check cannot catch it and you have to measure diversity separately. The last lines show the cycle loss: when F is the exact inverse of G the loss is 0.0000, with a random F it is 2.27, and training pushes the pair toward the first situation.

Diffusion models

Diffusion models take a different route. Instead of generating in one jump, they generate gradually by removing noise.

The forward process is fixed and has nothing to learn. Over T steps, add a little Gaussian noise to a clean image each time until nothing but static remains. A useful fact is that you do not have to simulate the steps one by one: with alpha_bar_t as the cumulative product of (1 - beta) over the steps, the noisy image at step t is

x_t = sqrt(alpha_bar_t) * x_0 + sqrt(1 - alpha_bar_t) * eps

so you can jump straight to any step. The training task is simple: show a network x_t and t, and ask it to predict the noise eps that was added. This is plain regression with a squared error loss, which is why diffusion training is stable.

The reverse process generates an image: start from pure noise, ask the network for its noise estimate, subtract a bit of it, add a little fresh randomness, and repeat for all T steps back to t = 0. Each step only needs to make the image slightly cleaner.

The example below does two things. First, it noises a 16x16 ring image and shows the signal fading as t grows. Second, it runs the reverse process on a toy where the data is exactly -2 or +2. For that special data the ideal noise predictor can be written down in one line, so we use it instead of a trained network. A real model learns an approximation of this function from data.

Python The forward process and a reverse process

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The table of weights is the heart of the forward process. At t = 0 the image is untouched (signal 1.000, noise 0.010); at t = 50 signal and noise are almost equal (0.721 against 0.693); at t = 99 the image is mostly noise (signal 0.280, noise 0.960). Knowing the true noise, we can recover the original exactly (True), and a network's job is to guess that noise without being told. In the reverse demo, we start from 2,000 random numbers and apply 100 reverse steps; they end up as 51% near -2 and 49% near +2, and none anywhere else. Pure noise has been turned into samples of the data distribution.

Production systems add three ingredients to this recipe: a large network (often a U-Net or a transformer, with time t as an input), conditioning on a text prompt through attention, and working in the compressed latent space of an autoencoder instead of on raw pixels, which is why this lesson follows the autoencoder lesson. Sampling is slower than a GAN because it needs many network calls, and a lot of research goes into reducing the number of steps.

Diffusion in practice: guidance, inpainting and fewer steps

The tiny experiments above hide how real image generators are driven. Four practical ideas turn the denoising loop into a product.

Text conditioning and guidance. A text-to-image model is a denoiser that also receives an embedding of your prompt (through cross-attention, see The Attention Mechanism). During training the prompt is dropped at random some of the time, so one network learns both an unconditional noise estimate eps_u and a conditional one eps_c. At sampling time classifier-free guidance extrapolates from the first to the second:

eps = eps_u + (1 + w) * (eps_c - eps_u)

A larger guidance scale pushes the sample harder toward whatever is specific to the prompt (text-to-image tools usually expose the scale 1 + w, with typical values around 5 to 10). The price is diversity: high guidance gives clean, on-prompt images that all look alike, and very high values give oversaturated, burnt-looking ones. A negative prompt simply replaces eps_u with the estimate for the text you want to avoid.

Python Classifier-free guidance on two overlapping classes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Class A and class B overlap on purpose: about 16% of true class-B data lies below 0, in the region where it could pass for class A. Plain conditional sampling (w = 0) reproduces this (18.6% here, within the error of a 100-step sampler). As w grows the samples move away from the other class, from 2.9% at w = 1 to 0.0% at w = 6, while the spread shrinks from 1.00 to 0.68. That is the guidance trade-off in one table: more obedience, less variety.

Inpainting. To fill a masked region, run the ordinary reverse loop but, at every step, overwrite the known pixels with a copy of the original image noised to the current level. The model then has to generate the missing part so that it is consistent with the visible part, because the visible part is always sitting in the picture at the right noise level. Outpainting extends the canvas the same way, and image-editing tools build on both. Models can also be trained directly with a mask as an extra input.

Fewer steps. Ancestral sampling needs a network call per step, often 1,000. DDIM and related samplers rewrite the reverse step as a deterministic update that can skip steps, and distillation can squeeze generation into a handful of calls. Latent diffusion (the idea behind Stable Diffusion) runs the whole process in the compressed latent space of an autoencoder, so each call handles a small grid instead of a full-resolution picture, and the decoder paints the final pixels.

Python Inpainting and DDIM on a two-point data set

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The data set has only two points, (-2, -2) and (2, 2), so the two coordinates are tied together. Unconditional sampling lands on the +2 side in 49% of samples. When the first coordinate is fixed to +2 by the inpainting trick, the missing coordinate comes out at +2 in all 3,000 samples, and fixing -2 gives -2 every time, so the model filled the hole consistently with what it could see. The DDIM loop reaches the right points even with 4 network calls, but that is because this toy is trivial and the noise predictor is exact: real images typically need 20 to 50 steps from a good sampler.

Other controls you will meet are ControlNet (a trainable copy of the encoder that adds a condition such as edges or pose), LoRA adapters (small fine-tunes for a style or a subject) and image-to-image (start the reverse process from a partly noised input image instead of from pure noise, so a low strength keeps the layout).

Which one to use?

VAE GAN Diffusion
Training Stable Unstable, needs care Stable
Sample quality Often blurry Sharp Very high
Sampling speed One pass One pass Many passes
Gives a density estimate Approximately (the ELBO) No Approximately
Typical uses Latent spaces, compression, anomaly scores Fast image synthesis, upscaling Text-to-image, audio, video

Language models are generative too, but they use the autoregressive recipe from Training a Tiny Language Model: generate one token at a time from a learned conditional distribution.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does a VAE use the reparameterization trick?

    A random draw blocks backpropagation. Moving the randomness into the separate input eps makes z a differentiable function of the encoder outputs.
  2. 2. What does the KL term in the VAE loss do?

    KL penalizes distributions that differ from N(0, 1). That is what keeps the latent space smooth and makes decoding prior samples work.
  3. 3. In a GAN, what is the generator trained to do?

    The generator receives gradients through the discriminator and learns to make fakes that the discriminator scores as real.
  4. 4. What is mode collapse?

    A generator can settle on a few outputs that currently fool the discriminator and stop covering the other modes of the data.
  5. 5. What does the network learn to predict in a standard diffusion model?

    Given the noisy image and the step t, the network estimates the noise. Subtracting that estimate step by step turns random noise into a sample.

Summary

  • Generative models learn a distribution you can sample from; VAEs, GANs and diffusion models are the three main image-era families.
  • A VAE encodes each input as a Gaussian, samples with z = mu + sigma * eps (the reparameterization trick), and adds a KL term that keeps the latent space close to N(0, 1), so decoding prior samples generates data.
  • A GAN trains a generator and a discriminator against each other; it is sharp and fast to sample, but unstable and prone to mode collapse.
  • GAN variants: DCGAN (transposed-convolution generator), cGAN (labels fed to both networks), CycleGAN (cycle-consistency loss, unpaired data) and StyleGAN (style vector injected at every layer); WGAN-style losses and diversity checks fight vanishing gradients and mode collapse.
  • A diffusion model adds noise on a fixed schedule, learns to predict that noise, and generates by denoising from pure noise step by step.
  • In practice diffusion is steered with classifier-free guidance (more obedience, less variety), inpainted by re-imposing the known pixels at every step, and sped up with DDIM-style samplers and latent-space denoising.
  • Pick by trade-off: VAEs for smooth latent spaces, GANs for fast sharp samples, diffusion for the best quality when you can afford many steps.

Next: Neural Networks in Reinforcement Learning.