47 · Diffusion models (DDPM)
Destroy an image with noise step by step, then learn to reverse it. A network predicts the noise added at each step and subtracts it to denoise — stable training plus high sample quality, which is why diffusion overtook GANs.
A diffusion model gradually DESTROYS data with Gaussian noise over T steps (a fixed forward process), then trains a network to predict and subtract that noise step by step — RESTORING a clean sample from pure noise. Destroy then restore.
Without this:
Diffusion is the engine of Stable Diffusion, DALL·E, Midjourney, and modern video/audio generation. Without it you can't reason about how today's best image models actually work.
GANs gave sharp samples but unstable training; VAEs gave stable training but blurry samples. Diffusion models finally get both — and the idea is almost shockingly simple: destroy, then restore.
Forward process (destroy). Take a clean data point x_0 (an image) and add a tiny bit of Gaussian noise. Repeat over T steps, each step a little noisier, until x_T is pure noise — all structure gone. This forward process has no learned parameters; it's a fixed corruption recipe set by a noise schedule. Beautifully, you don't have to step through it one noise-add at a time: there is a closed form that jumps straight to any step t,
x_t = √ᾱ_t · x_0 + √(1 − ᾱ_t) · ε, ε ∼ N(0, 1)
where ᾱ_t (alpha-bar) comes from the schedule and decreases from 1 toward 0 as t grows. At t=0, ᾱ=1: you get x_0 back. At t=T, ᾱ=0: you get pure noise ε. In between, x_t is a smooth blend of signal and noise — exactly what the cell below plots on a sine wave.
Reverse process (restore). Generation runs the forward process backwards. Start from pure noise x_T ∼ N(0,1) and step down to x_0, removing a little noise each step. But to undo a step you must know what noise was added — so we train a network ε_θ(x_t, t) to predict the noise ε that produced x_t (it takes the noisy input AND the timestep t). Subtract the predicted noise and you've taken one denoising step back toward clean data.
Training is dead simple. Pick a random data point and a random step t; draw noise ε ∼ N(0,1); form x_t with the closed form above; ask the network to predict ε; minimize ‖ε − ε_θ(x_t, t)‖². That's it — a plain regression of "what noise did I add?", with none of the adversarial instability of GANs. The cell below shows the forward process numerically; the scene animates it on a 2-D image.
Python (in browser)
Forward diffusion progressively noising a signal: a sine wave decays into pure noise as the cosine-schedule ᾱ_t falls from 1 to 0.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Forward process: xₜ = √ᾱₜ·x₀ + √(1−ᾱₜ)·ε. At t=0 the image is clean; at t=T it is pure noise.
DDPM pseudocode: training samples a step t and noise ε, forms x_t with the closed form, and regresses ε̂ = ε_θ(x_t,t) via MSE; sampling starts from pure noise and denoises T→1. The denoiser is a U-Net with a timestep embedding.
In a diffusion model, what does the network learn to predict at each step?
- The forward process adds Gaussian noise over T steps via the closed form x_t = √ᾱ_t·x_0 + √(1−ᾱ_t)·ε, with ᾱ_t from a schedule decreasing 1→0; it has no learned parameters.
- The reverse process trains a U-Net ε_θ(x_t,t) (with a timestep embedding) to predict the added noise, minimizing ‖ε − ε_θ(x_t,t)‖²; generation starts from pure noise and denoises step by step.
- Diffusion is an iterated denoising autoencoder — stable training (a plain regression, no adversary) AND high sample quality, which is why it overtook GANs for image synthesis.
Diffusion is the core of Stable Diffusion, DALL·E, Midjourney, Imagen, and modern video/audio generators — the dominant approach to high-quality image synthesis today.
If you remove it: You'd be unable to explain how today's best image models work, or why they replaced GANs — you'd know the buzzword but not the destroy-then-restore mechanism behind it.