48 · Latent & conditional diffusion
Stable Diffusion makes diffusion practical and controllable: diffuse in a VAE's compressed latent instead of pixels, inject a text prompt via cross-attention, and steer prompt adherence with classifier-free guidance.
Stable Diffusion runs diffusion in a VAE's compressed LATENT space (cheap), CONDITIONS the U-Net denoiser on a CLIP text embedding via cross-attention (controllable), and uses classifier-free guidance ε = ε_uncond + w·(ε_cond − ε_uncond) to dial how strongly the prompt is obeyed.
Without this:
Latent diffusion is what made text-to-image fast enough to run on consumer GPUs and controllable by a prompt — the exact recipe behind Stable Diffusion, DALL·E, and Midjourney.
Last lesson's diffusion ran directly on pixels — and that is expensive. A 512×512 RGB image is ~780k numbers, and you must run the denoiser dozens of times. Latent diffusion (the basis of Stable Diffusion) makes this practical with a clever move: don't diffuse in pixel space — diffuse in a compressed latent space.
A pretrained VAE (lesson 45) encodes the image down to a small latent (e.g. 64×64×4, ~50× fewer numbers). Run the entire diffusion process — forward noising and reverse denoising — inside that latent. Only at the very end does the VAE decoder turn the final clean latent back into a full-resolution image. Same diffusion math, a fraction of the compute.
That handles speed. Now control: how do we make it paint a specific prompt rather than a random image? Conditioning. A CLIP text encoder embeds the prompt ("a corgi astronaut") into a sequence of vectors. Those vectors are injected into the U-Net denoiser through cross-attention (Chapter 7): at each denoising step the image latent attends to the text tokens, letting the prompt steer what gets denoised into existence. The denoiser is no longer ε_θ(x_t, t) but ε_θ(x_t, t, text).
Finally, how strongly should the prompt be obeyed? Classifier-free guidance (CFG) is the dial. At each step the network predicts the noise twice — once with the prompt (ε_cond) and once without it (ε_uncond) — and combines them:
ε = ε_uncond + w · (ε_cond − ε_uncond)
The term (ε_cond − ε_uncond) is the direction the prompt pulls. The guidance scale w amplifies it: w=1 is the plain conditional prediction; larger w pushes further toward (and past) the prompt — stronger adherence and saturation, at the cost of diversity. The cell below shows exactly this interpolation: as w grows, the result marches toward and beyond ε_cond.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The Stable Diffusion pipeline in diffusers: CLIP text encoder → U-Net latent denoiser with cross-attention → VAE decoder; StableDiffusionPipeline.from_pretrained then pipe(prompt, guidance_scale=7.5, num_inference_steps=30).
What does the classifier-free guidance scale w control?
- Latent diffusion (Stable Diffusion) runs the whole diffusion process inside a VAE's compressed latent and decodes only at the end — slashing compute versus pixel-space diffusion.
- Conditioning injects a CLIP text embedding into the U-Net denoiser via cross-attention so the prompt steers generation; the denoiser becomes ε_θ(x_t, t, text).
- Classifier-free guidance ε = ε_uncond + w·(ε_cond − ε_uncond) dials prompt adherence: larger w pushes harder toward the prompt (default ~7.5) at the cost of diversity.
Latent + conditional diffusion is the exact recipe behind Stable Diffusion, DALL·E, and Midjourney — every text-to-image tool you use composes a VAE, U-Net, cross-attention, CLIP, and diffusion.
If you remove it: You'd understand raw diffusion but not how it became fast, prompt-controllable text-to-image — the practical leap that put generative imaging in everyone's hands.