Backpropagation and Automatic Differentiation

Backpropagation is the chain rule, applied efficiently to a computational graph — a directed acyclic graph where each node is a primitive operation and edges carry intermediate values. To compute $\nabla_\theta L$, we start from the loss at the top and walk the graph in reverse, multiplying by each operation's local Jacobian as we go. The cost is roughly twice the forward pass, regardless of how many parameters there are.

Modern auto-diff (PyTorch, JAX, TensorFlow) implements this as two modes. Forward-mode computes $J\mathbf{v}$ (Jacobian-vector product) by walking the graph forward while carrying tangent vectors. Reverse-mode (= backpropagation) computes $\mathbf{u}^\top J$ (vector-Jacobian product) by walking backward. When the output is scalar (a loss) and inputs are high-dimensional (parameters), reverse mode is overwhelmingly cheaper — one backward pass replaces millions of forward-mode passes.

A critical insight: we never materialize the full Jacobian. For a neural network with millions of parameters, the Jacobian is an unmanageable matrix. Instead, backprop computes Jacobian-vector products implicitly, layer by layer. For a matrix multiply $\mathbf{y} = W\mathbf{x}$, the gradient contribution is $\nabla_W L = \delta \mathbf{x}^\top$ and $\nabla_{\mathbf{x}} L = W^\top \delta$ — each is a single matrix multiply, never an explicit Jacobian.

Auto-diff handles arbitrary computations, not just neural networks. Any differentiable function expressible as a composition of primitives (add, multiply, log, exp, matmul, etc.) is automatically differentiable. This is why modern ML frameworks can compute gradients of non-standard architectures, custom losses, variational objectives, and even implicit functions via the implicit function theorem.

Two practical pitfalls: vanishing gradients (derivatives of sigmoids/tanhs shrink near saturation, and products of many small numbers collapse to zero) and exploding gradients (products of many large numbers blow up). Techniques like ReLU activations, residual connections, and gradient clipping address these directly. Understanding how gradients propagate — and where they die — is essential for training deep networks.

Worked example — forward vs reverse mode cost: Consider $f : \mathbb{R}^{1000} \to \mathbb{R}$. Forward mode needs 1000 passes (one per input) to fill the Jacobian; reverse mode needs just 1 backward pass. Conversely, for $f : \mathbb{R} \to \mathbb{R}^{1000}$, forward mode wins with 1 pass versus 1000. This asymmetry is why neural-net training (millions of inputs, one scalar loss) always uses reverse mode.