5 · Activation functions: sigmoid, tanh, ReLU + variants
A tour of the six activations that matter — what they look like, how their gradients behave, and which to pick for which task.
Activations inject non-linearity — without them stacked layers collapse to a single linear transform. The choice matters: sigmoid saturates, ReLU dies, Leaky ReLU and GELU fix the corner cases.
Without this:
Pick the wrong activation and your deep net either vanishes (sigmoid) or dies (vanilla ReLU).
An activation function sits between each linear layer and introduces non-linearity into the network. Without it, the entire network — no matter how many layers — would reduce to a single matrix multiplication: W_n × W_{n-1} × … × W_1 × x. You'd have the expressive power of a logistic regression.
The history of activations is almost a history of deep learning itself:
- 1943–1986 — step/sigmoid (the biological metaphor: neuron fires or doesn't)
- 2011 — ReLU (Glorot et al.) makes deep networks trainable: simple, fast, gradient-friendly
- 2012 — AlexNet proves ReLU + GPU + Dropout works at ImageNet scale
- 2016+ — ELU, SELU push beyond dying ReLU
- 2018+ — GELU (used in BERT, GPT, ViT) becomes the modern transformer default
Each activation is a trade-off between expressiveness, gradient health, and computational cost. This lesson maps all six, shows their derivative story, and gives you the selection rubric used by practitioners.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The dying ReLU problem — and why Leaky ReLU / ELU / GELU fix it
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Activation selection rubric — which function for which layer and task
What is the gradient of ReLU when x < 0?
- Without a non-linear activation, stacked layers collapse to a single linear transform — no universal approximation.
- ReLU (derivative = 1 for z > 0) is the default hidden-layer activation: fast, gradient-friendly, unlocked deep learning in 2012.
- Dying ReLU occurs when z < 0 for all training examples → gradient = 0 → neuron never updates. Leaky ReLU, ELU, and GELU fix this with non-zero gradients for negative inputs.
- Selection rubric: ReLU = hidden default; GELU = transformer default; sigmoid = binary output only; softmax = multiclass output; never sigmoid in hidden layers.
Every PyTorch `nn.Linear → nn.ReLU` block uses ReLU; GELU dominates Transformers; sigmoid + BCE is the standard binary-classification head.
If you remove it: You'd default to sigmoid everywhere and wonder why your 10-layer net converges to random chance — vanishing gradient is invisible until you understand activations.