2 · How an ANN trains: forward, loss, backward, update
The four-step training loop every neural net follows — by hand, in NumPy, then as 5 lines of PyTorch.
Training a neural net = forward pass to compute predictions, loss to measure errors, backward pass (chain rule) to compute gradients, optimizer step to update weights — repeat.
Without this:
Training is a black-box `model.fit()` instead of a precise, debuggable algorithm.
Every neural network training loop, regardless of framework or architecture, is the same four steps repeated until loss converges:
- Forward pass — compute predictions from inputs through all layers
- Loss — measure how wrong the predictions are (e.g. MSE, cross-entropy)
- Backward pass (backpropagation) — compute the gradient of loss w.r.t. every weight using the chain rule
- Optimizer step — nudge each weight in the direction that decreases loss:
w ← w - lr * grad
This loop is what model.fit() in Keras, optimizer.step() in PyTorch, and every JAX training step encapsulate. Once you've done it by hand, every framework feels familiar.
We'll use a 2-input → 2-hidden → 1-output network with sigmoid activations — small enough to trace by hand, large enough to be a real MLP.
The 4-step training loop — the algorithm every framework implements
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
PyTorch read-along — the same 4-step loop in 5 lines of nn.Module
Backpropagation and Automatic Differentiation — the mathematical foundation of step 3
What does `loss.backward()` in PyTorch do?
- Every training loop is 4 steps: forward pass → loss → backward pass (chain rule) → weight update. Repeat.
- The backward pass propagates `dL/dW` for every weight using the chain rule — stacking layers means chaining derivatives.
- PyTorch's `loss.backward()` automates step 3; `optimizer.step()` automates step 4 — but the algorithm is identical to the manual version.
- The vanishing gradient problem (sigmoid saturates at 0.25 max gradient) limits deep sigmoid networks — solved by ReLU (lesson 4).
Every PyTorch / TensorFlow / JAX training loop is exactly these 4 steps. Once you see it manually, every framework feels familiar.
If you remove it: Training is a black-box `model.fit()` instead of a precise, debuggable algorithm — you can't diagnose why loss plateaus or gradients explode.