44 · Autoencoders
An encoder squeezes input through a narrow bottleneck and a decoder rebuilds it. Forcing reconstruction through a smaller code makes the network learn the data's essential structure — the seed of every generative model in this chapter.
An autoencoder learns to copy its input to its output through a narrow bottleneck — and because the code is smaller than the input, the only way to reconstruct well is to learn a compressed representation of the data's true structure.
Without this:
Autoencoders are the conceptual ancestor of VAEs, diffusion denoisers, and the latent space of Stable Diffusion. Skip them and the rest of the chapter has no foundation.
Every model so far had a label to predict. An autoencoder has none — its target IS its input. It learns to copy what it is given, but with a deliberate obstacle in the middle, and that obstacle is the whole point.
The network has two halves. The encoder maps the input x down to a small code z (also called the latent or bottleneck). The decoder maps that code back up to a reconstruction x̂, the same shape as x. Training minimizes the reconstruction loss ‖x − x̂‖² — make the output match the input.
If the code were as large as the input, the network could cheat by copying straight through and learn nothing. The magic is the bottleneck: the code is much smaller than the input. To squeeze x into a handful of numbers and still rebuild it, the encoder must discover the data's essential structure — which directions of variation actually matter — and throw away noise and redundancy. The latent code is a learned, compressed summary of the input.
This is a non-linear cousin of PCA (Chapter on dimensionality reduction). In fact, a linear autoencoder with a squared-error loss recovers exactly the PCA subspace — the cell below makes that concrete. Swap in non-linear layers and the autoencoder can capture curved, non-linear structure that PCA cannot.
What are autoencoders good for? Compression (the code is a compact representation), denoising (train it to map a corrupted input back to the clean one — the denoising autoencoder), anomaly detection (inputs unlike the training data reconstruct poorly, so a high reconstruction error flags outliers), and representation learning (the latent code becomes a useful feature vector for downstream tasks). And crucially, the encoder→bottleneck→decoder shape recurs throughout this chapter: the VAE adds probability to the bottleneck, diffusion is an iterated denoising autoencoder, and Stable Diffusion literally diffuses inside an autoencoder's latent space.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
A convolutional autoencoder in PyTorch: encoder squeezes images to a 16-D code, decoder mirrors it back, MSE reconstruction loss; the denoising variant maps a corrupted input to the clean image.
Why does the bottleneck (a code smaller than the input) force an autoencoder to learn something useful?
- An autoencoder reconstructs its own input through a narrow bottleneck; minimizing ‖x − x̂‖² forces the encoder to learn a compressed representation of the data's essential structure.
- A linear autoencoder with squared error recovers the PCA subspace; non-linear layers extend this to curved structure. Uses: compression, denoising, anomaly detection, representation learning.
- A plain autoencoder is NOT a generator — sampling a random latent code usually decodes to garbage because the latent space has holes; the VAE fixes this next lesson.
Autoencoders power dimensionality reduction, image denoising, and anomaly detection in production; their encoder→bottleneck→decoder shape is the backbone of VAEs, diffusion denoisers, and Stable Diffusion's latent space.
If you remove it: You'd see VAEs and diffusion as unrelated tricks instead of variations on one idea — squeezing data through a learned bottleneck and rebuilding it.