11 · Dropout: stochastic regularisation
Dropout randomly zeroes activations during training — forcing redundant representations and defeating overfitting without a single extra parameter.
Dropout randomly zeroes out a fraction of activations during training — forcing the network to learn redundant features and preventing co-adaptation.
Without this:
Deep nets memorize training data; dropout is one of the simplest regularizers that actually works.
Deep networks have millions of parameters and will happily memorise the training set if you let them. The textbook fix is more data, but the practical fix is regularisation — penalising the model for being too clever about individual examples.
Dropout (Srivastava et al., 2014) is gloriously blunt: at each training step, randomly set each neuron's output to zero with probability p. The remaining neurons have their outputs scaled up by 1/(1-p) (inverted dropout) so the expected magnitude matches inference time.
Why does this work?
- Breaks co-adaptation: neurons can't rely on specific partners always being present, so each learns more independently useful features.
- Implicit ensemble: with
Nneurons, each training step trains one of2^Npossible subnetworks. Inference averages across all of them. - No extra parameters: dropout adds zero learnable parameters — it's pure stochasticity.
At inference time, dropout is disabled and all neurons are active. Because we used inverted dropout during training, no adjustment is needed at test time — the expected activation is already correct.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Mental model: dropout as training an ensemble of 2^N subnetworks
PyTorch read-along: nn.Dropout — train mode vs eval mode
Why does inverted dropout scale activations by `1/(1-p)` during training?
- Dropout zeros a fraction p of activations at each training step — each step trains a different random subnetwork.
- Inverted dropout scales survivors by 1/(1-p) during training so inference works unchanged — no scaling adjustment needed at test time.
- ALWAYS call model.eval() before inference — forgetting leaves dropout active and produces noisy, unreliable predictions.
- Typical values: p=0.5 for FC heads (AlexNet style), p=0.1-0.3 for transformer layers. Never apply dropout after the output layer.
Dropout p=0.1-0.3 is standard in transformers; p=0.5 was AlexNet's setting for CNN heads; nearly every PyTorch architecture uses some dropout.
If you remove it: Your model memorises the training set, achieves near-zero train loss, and generalises poorly — validation loss climbs while train loss plummets.