2 · Cost function & gradient descent
The universal optimizer — take the slope of the loss, step opposite to it, repeat until convergence.
Gradient descent is how almost every modern ML model learns — take the slope of the loss, step opposite to it, repeat.
Without this:
Without gradient descent you can't optimise neural networks (no closed form exists) — and the OLS closed form breaks at scale anyway.
In the previous lesson we found the best line by deriving the OLS closed form — setting the gradient to zero and solving analytically. That works beautifully for linear regression with moderate data, but it has hard limits:
- The normal equation requires inverting an (n × n) matrix — O(n³) time. At 10 million rows, that's infeasible.
- Neural networks have millions of parameters with no closed-form solution — every weight must be found iteratively.
Gradient descent is the answer. The key insight is geometric: the gradient ∇L at a point in parameter space tells you which direction makes the loss increase fastest. Step in the opposite direction, and you reduce the loss. Repeat, and you spiral down the loss surface toward the minimum.
For MSE loss with parameters (w, b), the gradients are:
∂L/∂w = −(2/n) · Σ(yᵢ − ŷᵢ) · xᵢ
∂L/∂b = −(2/n) · Σ(yᵢ − ŷᵢ)
The update rule at each iteration t is:
wₜ₊₁ = wₜ − η · ∂L/∂w
bₜ₊₁ = bₜ − η · ∂L/∂b
where η (eta) is the learning rate — a hyperparameter that controls step size. Getting η right is the most important practical skill in training ML models.
Python (in browser)
Watch w and b converge toward 2.5 and 1.0 over 200 iterations. The MSE column shows the loss falling — that's learning.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Semi-log y-axis reveals the rapid early drop and asymptotic approach. Three learning rates, three stories: too slow, just right, diverges.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The characteristic loss curve: a steep drop in the first few dozen iterations, then a long plateau. The plateau means the model is near its minimum — further iterations return diminishing improvements.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Mini-batch GD is the industry default — batch size is tuned, not chosen arbitrarily. Start with 32 or 64.
If gradient descent's loss oscillates between high and low values without converging, the most likely cause is:
Gradient Descent — the MML chapter covering the update rule, convergence conditions, and the geometric intuition of the loss surface.
- The gradient ∂L/∂w tells you which direction increases the loss — step opposite to it: w ← w − η·∂L/∂w.
- Learning rate η is the most critical hyperparameter: too small → slow convergence; too large → diverges or oscillates.
- Batch GD uses all data per step (stable, slow). SGD uses one sample (fast, noisy). Mini-batch (32–256 samples) is the industry default.
- Adam, RMSprop, AdaGrad are adaptive variants of GD — the core step is identical; they add per-parameter learning rate scaling.
Every neural network is trained by gradient descent (with Adam etc. as variants). XGBoost performs gradient descent in function space. SGD with momentum is the universal default for deep learning.
If you remove it: You'd be stuck with closed-form-only models — no neural nets, no online learning, no models with millions of parameters.