8 · Optimizers: SGD → Momentum → AdaGrad → RMSProp → Adam
Build Adam from scratch, visualize momentum on a saddle surface, and understand why Adam is the modern default.
Optimizers refine vanilla SGD: momentum adds velocity, AdaGrad scales per-parameter, RMSProp fixes AdaGrad's decay, Adam combines momentum + RMSProp. Adam is the modern default.
Without this:
You'd train with vanilla SGD and learning rates that work for SOME params but not others.
Vanilla SGD updates every weight with the same learning rate: θ ← θ - lr * gradient. Simple and correct in theory — but in practice it has two problems:
- Same learning rate for all parameters: weights connected to rare features (in NLP: rare words) need larger updates; common features need smaller ones.
- No memory: each step ignores the direction of previous steps. The optimizer zigzags in high-curvature directions instead of smoothly heading downhill.
The evolution of optimizers solved both:
- SGD + Momentum — adds a "velocity" vector that accumulates direction. Fewer oscillations, faster convergence on smooth loss surfaces.
- AdaGrad — adapts lr per parameter based on accumulated squared gradients. Parameters that receive large gradients get smaller lr. Problem: lr decays to ~0 over time.
- RMSProp — uses an exponential moving average of squared gradients instead of a sum. Fixes AdaGrad's decaying lr.
- Adam (Adaptive Moment Estimation) — combines momentum (first moment) with RMSProp (second moment) + bias correction. The modern default for nearly every network.
- AdamW — Adam + proper weight decay. Current default for Transformers.
The optimizer family tree: SGD → Momentum → AdaGrad → RMSProp → Adam → AdamW
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Optimizer selection rubric: when to use Adam, AdamW, SGD+Momentum, RMSProp
Gradient Descent — the mathematical foundation every optimizer builds on
Adam's β₁ controls the moving average of...?
- SGD is the foundation; momentum adds velocity (reduces oscillations); AdaGrad adapts per-parameter lr; RMSProp fixes AdaGrad's decay; Adam = momentum + RMSProp + bias correction.
- Adam's update rule: `θ -= lr * (m/√v)` where m is gradient EMA (β₁=0.9) and v is squared-gradient EMA (β₂=0.999), both bias-corrected.
- Adam = modern default for most tasks; AdamW = modern default for Transformers; SGD+Momentum still competitive for large-scale CV.
- Bias correction (dividing by `1 - β^t`) is critical in the first ~100 steps to correct the zero-initialization bias of m and v.
Every PyTorch training script starts with `optim.Adam(model.parameters(), lr=1e-3)`. Knowing what's INSIDE the optimizer lets you debug 'why isn't my model converging'.
If you remove it: You'd treat the optimizer as a magic knob — unable to diagnose why loss plateaus, why the model diverges in early steps, or why different parameters need different learning rates.