Higher-Order Derivatives

Taking the derivative of a derivative gives higher-order derivatives. In 1D, $f''(x)$ measures curvature — how fast the slope is itself changing. Large positive $f''$ means the function curves up sharply (a narrow bowl), small $f''$ means gentle curvature (a wide bowl), and negative $f''$ means curving down (a hill).

For multivariable $f : \mathbb{R}^n \to \mathbb{R}$, the second-order object is the Hessian matrix $H_f \in \mathbb{R}^{n \times n}$ with entries $H_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j}$. For sufficiently smooth $f$, mixed partials commute (Clairaut's theorem): $\frac{\partial^2 f}{\partial x_i \partial x_j} = \frac{\partial^2 f}{\partial x_j \partial x_i}$, so the Hessian is symmetric.

The Hessian characterizes critical points. At a point where $\nabla f = \mathbf{0}$: if $H$ is positive definite (all eigenvalues > 0), it is a local minimum; if negative definite, a local maximum; if indefinite (mixed signs), a saddle point; if singular, the test is inconclusive and you need higher-order analysis. The eigenvalues of the Hessian give the curvatures along the principal axes of the quadratic form.

The condition number of the Hessian — the ratio $\lambda_{\max} / \lambda_{\min}$ — governs how hard the optimization problem is. A high condition number means the loss landscape has very elongated valleys: gradient descent zigzags because it can't find the long narrow direction. This is exactly why Newton's method $\mathbf{x}_{k+1} = \mathbf{x}_k - H^{-1}\nabla f$ converges faster: it rescales by the inverse Hessian, making the effective landscape isotropic.

In ML, the Hessian drives second-order optimization (Newton, quasi-Newton methods like L-BFGS, natural gradient descent), Laplace approximations of Bayesian posteriors (approximate a posterior by a Gaussian with covariance $H^{-1}$), and sharpness-based generalization (flat minima with small Hessian eigenvalues tend to generalize better than sharp ones). Storing and inverting Hessians is expensive ($O(n^2)$ storage, $O(n^3)$ inversion), which is why most deep learning uses first-order methods or stochastic approximations.

Worked example — classify critical point of $f(x,y) = x^2 + xy + y^2$: Gradient $\nabla f = (2x + y,\; x + 2y)$ vanishes only at $(0,0)$. Hessian $H = \begin{pmatrix}2 & 1 \\ 1 & 2\end{pmatrix}$ has eigenvalues $\lambda = 1, 3$ (both positive). Since $H$ is positive definite, the origin is a strict local minimum, matching the fact that $f$ is a convex quadratic form.