3 · Backpropagation: the chain rule, mechanized
Trace every gradient from output to input, verify against finite differences, and understand what PyTorch's autograd computes.
Backprop is the chain rule applied recursively from output layer back to input, accumulating partial derivatives at each step.
Without this:
PyTorch's `.backward()` stays magic instead of a clean algorithm.
Backpropagation is the algorithm that makes deep learning scale. The idea is elegantly simple: the loss is a composition of functions (layers), so its gradient w.r.t. every weight is computed via the chain rule applied in reverse — from the output layer back to the input.
Each layer in a neural network is a node in a computational graph. Every node:
- Stores its forward-pass output value
- Knows its "local gradient" function (how its output changes w.r.t. its inputs)
During backprop, we pass a gradient signal backwards through the graph: each node multiplies the incoming gradient by its local gradient and passes the product upstream. By the time the signal reaches the first layer, every dL/dW has been computed.
This lesson traces the backprop algorithm by hand on the same 2-2-1 network from lesson 2, verifies each gradient numerically, and shows the identical result from PyTorch's autograd.
Chain rule recap — scalar form, computational graph, vector form
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
PyTorch autograd read-along — same gradients as the hand-computed version
Backpropagation and Automatic Differentiation — the formal treatment of the computational graph and chain rule
If your analytical gradient differs from the numerical gradient by 10%, what should you suspect?
- Backprop = chain rule applied in reverse through the computational graph; each node multiplies the incoming gradient by its local gradient.
- The gradient of a weight W in layer L is: `dL/dW = upstream_gradient @ local_input^T`.
- Numerical gradient check `(L(w+h) - L(w-h)) / 2h` is the gold standard for verifying custom backward passes — relative error should be < 1e-5.
- PyTorch's `loss.backward()` computes the same gradients via a dynamic computational graph built during the forward pass — call it once per forward pass.
Backprop is the engine that lets us train networks with billions of parameters. Without it, you'd be stuck doing forward-pass-only inference on hand-tuned weights.
If you remove it: Every custom layer (attention head, graph convolution, normalizing flow) requires a correct backward pass — without understanding backprop, you can't implement or debug these.