Skip to content
Tutorials Center
My learning

Topics

Lesson 41 of 53 ยท Beyond Text

How Image Generation Works

Understand diffusion models: adding and removing noise, latent space, text conditioning, classifier-free guidance, image prompting and editing.

Language models write one token after another. Most image generators work differently: they start from pure random noise and clean it up step by step until a picture appears. These are diffusion models, and they power most of the text-to-image tools you have seen. Other families exist (GANs, autoregressive image models), but diffusion is the main idea to understand.

You cannot train or run a diffusion model in the browser, so most of this lesson is conceptual. What you can run is the easy half of the process, adding noise to an image, which shows exactly what the model learns to undo. For text generation, compare with How LLMs Are Trained.

From noise to picture

Start with noise Pure random pixels or latents.
Denoise a little The model predicts the noise to remove.
Repeat 20 to 50 steps, each cleaner.
Image Steered by your text prompt.
1 / 2

The forward process: destroying an image

Training starts with real images. For each one, we add Gaussian noise in many small steps until nothing but static is left. This forward process needs no learning. A convenient formula jumps straight to any step t:

TEXT
x_t = sqrt(a_t) * x_0 + sqrt(1 - a_t) * noise

x_0 is the clean image, noise is random normal values, and a_t (alpha-bar) shrinks from almost 1 at the start to almost 0 at the end. At small t the picture is barely changed; at the last step it is pure noise.

Python Forward noising: an image dissolving into static

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At t = 0 the circle is clear; by t = 500 it is faint; at t = 999 the signal share is about 0.006 and the image is essentially noise.

The reverse process: what the model learns

Training teaches a neural network (traditionally a U-Net, increasingly a transformer, see The Transformer Block) one small job: given a noisy image x_t, the step t and a text description, predict the noise that was added. The loss is simply the mean squared error between the predicted and the true noise. Because the model sees every noise level, it learns both coarse structure (at high noise) and fine texture (at low noise).

To generate, you start from random noise and loop:

  1. Predict the noise in the current image.
  2. Subtract a fraction of it (how much depends on the sampler, such as DDIM or DPM-Solver) and, in some samplers, add a little fresh noise.
  3. Move to the next, less noisy step.

Fewer steps are faster but can lose detail. Modern distilled models need only a handful. The random starting noise is the seed: the same seed, prompt and settings give the same image, while a new seed gives a new variation.

flowchart LR
    A["Random noise"] --> B["Denoiser (U-Net or transformer)"]
    P["Text embedding"] --> B
    B --> C["Slightly cleaner image"]
    C -->|"repeat 20 to 50 times"| B
    C --> D["Final image"]

The same denoiser runs at every step; the text embedding steers each one.

Latent space: working on a compressed image

Denoising a full 1024 by 1024 image is expensive. Latent diffusion (the approach behind Stable Diffusion and many others) first compresses images with an autoencoder. The encoder maps an image to a much smaller grid of numbers called the latent; the decoder turns a latent back into pixels. Diffusion then runs in latent space, often 8 times smaller in each direction (64 times fewer positions), and the decoder produces the final image at the end. This is why generation is feasible on consumer hardware.

Text conditioning and guidance

How does the text control the picture? A text encoder (a transformer, often CLIP-style or a language model) turns the prompt into embeddings. The denoiser reads them through cross-attention, the same query-key-value mechanism from Attention from Scratch: image positions are the queries, text tokens provide keys and values.

Classifier-free guidance (CFG) strengthens the effect. At each step the model predicts noise twice, once with the prompt and once with an empty prompt, and then pushes the result away from the empty prediction:

TEXT
noise = noise_uncond + scale * (noise_cond - noise_uncond)

A scale of 1 follows the prompt loosely; around 5 to 8 is a common range; very high values give oversaturated, harsh, less varied images. The next example shows the arithmetic with two tiny vectors.

Python Classifier-free guidance on toy noise predictions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At scale 0 the prompt is ignored, at 1 you get the plain conditional prediction, and beyond that the push away from "no prompt" grows linearly, which is why extreme values distort images.

Prompting images

Image prompts work best as a clear description of the picture, not as instructions to a chatbot:

  • Subject first, then setting, style, lighting, camera or medium, and mood: "a lighthouse on a cliff at dusk, watercolor, soft warm light, wide shot".
  • Be concrete. Specific nouns beat adjectives like "beautiful".
  • Negative prompts (supported by some tools) list what to avoid, using the empty-prompt side of guidance.
  • Counting, small text and hands are traditional weak spots; newer models are better but still imperfect.
  • Fix the seed when you compare prompts, so only the prompt changes. See Prompt Debugging for the same iterate-and-compare habit.

Editing and control

The same machinery supports more than text-to-image:

  • Image-to-image: start from a noised version of your picture instead of pure noise. The amount of noise (often called strength) sets how much changes.
  • Inpainting and outpainting: regenerate only a masked region, or extend beyond the borders.
  • Control inputs: edge maps, depth maps or poses guide the layout (ControlNet-style adapters).
  • Personalization: small LoRA files teach a model a style or subject from a few images.
  • Instruction editing: models that take an image plus an instruction ("make the sky purple").

Read-only sketch of a typical library call (needs a GPU and downloaded weights):

Python
# Not runnable here.
pipe = SomeDiffusionPipeline.from_pretrained("model-name")
image = pipe(prompt="a lighthouse at dusk, watercolor",
             num_inference_steps=30, guidance_scale=7.0,
             generator=make_generator(seed=42)).images[0]
image.save("lighthouse.png")

Risks and limits

Generated images raise the questions from Bias, Privacy, Copyright and Responsible Use: training-data licensing, likeness and deepfakes, biased defaults for people and professions, and the need to label synthetic media. Watermarking and provenance standards (such as C2PA metadata) help but can be removed, so do not treat them as proof.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the denoising network predict during training?

    The loss compares predicted noise with the real noise added in the forward process.
  2. 2. Why does latent diffusion run on a compressed representation?

    Working on a latent that is much smaller than the image cuts the compute needed per step.
  3. 3. What does a very high guidance scale usually cause?

    Large scales push far from the unconditional prediction, which exaggerates the prompt and distorts the image.
  4. 4. Same prompt, settings and seed on the same model. What do you expect?

    The seed fixes the starting noise, so the process is repeatable on the same software and hardware.

Summary

  • Diffusion models learn to remove noise; generation runs the denoiser from pure noise over many steps.
  • The forward process adds noise by a closed formula; the network is trained to predict that noise.
  • Latent diffusion works in a compressed space, then decodes to pixels.
  • Text steers generation through embeddings, cross-attention and classifier-free guidance.
  • Prompts describe the picture; seeds make results repeatable; editing reuses the same process.

Next: Audio, Video and Multimodal Models.