Partial Derivatives and Gradients

For a function $f : \mathbb{R}^n \to \mathbb{R}$ of several variables, the partial derivative $\frac{\partial f}{\partial x_i}$ is the rate of change of $f$ as $x_i$ varies while all other variables are held fixed. Computationally, you differentiate $f$ treating every variable but $x_i$ as a constant.

The gradient $\nabla f(\mathbf{x})$ packages all the partial derivatives into a single vector: $\nabla f = \left(\frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_n}\right)^\top$. The gradient has a beautiful geometric interpretation: it points in the direction of steepest ascent, and its magnitude is the rate of increase in that direction. The negative gradient points toward steepest descent — which is why gradient descent works.

The directional derivative in direction $\mathbf{u}$ (a unit vector) is $D_{\mathbf{u}} f = \nabla f \cdot \mathbf{u}$ — the projection of the gradient onto $\mathbf{u}$. This lets us read off two facts: the gradient direction gives the maximum rate of increase ($\|\nabla f\|$), and any direction perpendicular to the gradient gives zero instantaneous change (you are moving along a level set).

At a smooth interior extremum, the gradient vanishes: $\nabla f(\mathbf{x}^*) = \mathbf{0}$. Such points are called critical points or stationary points and come in three flavors: local minima, local maxima, and saddle points (minimum in some directions, maximum in others). In high-dimensional deep learning landscapes, saddle points are overwhelmingly more common than local minima, a fact that reshapes how we think about optimization.

In ML, every training step is gradient-based: compute $\nabla_{\mathbf{w}} L(\mathbf{w})$, step in its negative direction. Variants of this idea — SGD, Adam, RMSprop — differ only in how they denoise or adaptively rescale the gradient. A working intuition for gradients is arguably the single most important mathematical skill for a practicing ML engineer.