46 · Generative adversarial networks (GANs)
Two networks duel: a generator forges samples and a discriminator tries to catch them. Their minimax game produces strikingly sharp images — but the adversarial training is notoriously unstable, prone to mode collapse, and offers no likelihood.
A GAN pits a GENERATOR (turns noise into fake samples) against a DISCRIMINATOR (judges real vs fake) in a minimax game; as the discriminator improves, it pushes the generator to make samples indistinguishable from real data — yielding sharp images but unstable training.
Without this:
GANs defined the sharp-image era of generative AI (StyleGAN faces, super-resolution) and frame the central trade-off — sample quality vs training stability — that diffusion later resolved.
VAEs gave us a sampleable generator but blurry images. GANs swing hard the other way: razor-sharp samples, at the price of training that can fall apart. The idea is a game between two networks.
- The generator
Gtakes a random noise vectorz ∼ N(0,1)and outputs a fake sampleG(z)(an image, say). It has never seen real data directly — it only learns through the critic. - The discriminator
Dis a binary classifier that takes a sample and outputs the probability it is real (from the training set) rather than fake (fromG).
They train adversarially, in opposition. D is trained to correctly label real samples as real and fakes as fake. G is trained to fool D — to make D(G(z)) close to 1, as if its forgeries were real. This is the minimax objective: D maximizes its accuracy while G minimizes it. Picture a forger and a detective: the detective gets sharper at spotting fakes, which forces the forger to produce better counterfeits, which forces the detective to sharpen again. At the ideal equilibrium, G's samples are indistinguishable from real data and D is reduced to guessing (output ≈ 0.5).
Why sharp? Unlike the VAE, there is no pixel-wise reconstruction loss rewarding "safe average" outputs. The only signal is "does this fool the critic?" — and a blurry image is easy to flag as fake, so G is pushed toward crisp, realistic detail.
The catch is the cost of that sharpness. Adversarial training is a moving target: both networks change every step, so there is no fixed loss landscape descending to a minimum — it is an equilibrium-seeking game that can oscillate or diverge. The classic failure is mode collapse: G discovers a few outputs that reliably fool D and produces only those, ignoring the diversity of the real data (e.g. generating the same handful of faces). And a GAN gives you no likelihood — no p(x) you can evaluate — so you can't even cleanly measure how well it fits the data. The toy cell below runs this duel in 1-D and watches the generator chase the real distribution, oscillating as it goes.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
A DCGAN in PyTorch: a generator maps noise to a fake image, a discriminator outputs P(real), and the adversarial loop alternates training the two opposed objectives; lists key variants (WGAN-GP, StyleGAN, CycleGAN).
In a GAN, how does the generator learn — given it never sees real data directly?
- A GAN trains a generator (noise → fake) against a discriminator (real vs fake) in a minimax game; the generator learns only through the discriminator, pushing D(G(z)) → 1 to fool it.
- With no pixel-wise reconstruction loss, GANs produce strikingly sharp samples — but adversarial training is unstable, prone to mode collapse, and gives no likelihood p(x).
- GAN (sharp, unstable, no likelihood) and VAE (blurry, stable, likelihood) are mirror trade-offs; diffusion later achieves both stable training AND high quality.
GANs power photorealistic face synthesis (StyleGAN), super-resolution, image-to-image translation (CycleGAN), and data augmentation; their adversarial-loss idea reappears far beyond image generation.
If you remove it: You'd miss the sharp-image era of generative AI and the quality-vs-stability trade-off that motivated diffusion — the model family the rest of the chapter builds on.