Gradient Descent

Gradient descent is the simplest and most-used continuous optimization algorithm: to minimize a differentiable $f(\mathbf{x})$, start at some $\mathbf{x}_0$ and iteratively update $\mathbf{x}_{k+1} = \mathbf{x}_k - \eta \nabla f(\mathbf{x}_k)$, where $\eta > 0$ is the learning rate. Each step moves in the direction of steepest descent — the negative gradient — by an amount scaled by $\eta$.

The learning rate is the most important hyperparameter. Too small, and convergence is glacial. Too large, and the algorithm may overshoot and diverge. For convex quadratics, the optimal $\eta$ is related to the Hessian's eigenvalues: $\eta \leq 2/\lambda_{\max}$ for stability, and the convergence rate depends on the condition number $\kappa = \lambda_{\max}/\lambda_{\min}$. Ill-conditioned problems (large $\kappa$) converge slowly.

Practical variants include momentum ($\mathbf{v}_{k+1} = \beta \mathbf{v}_k + \nabla f$, step in $\mathbf{v}$), which accelerates along consistent directions and damps oscillation. Adam adapts per-parameter learning rates by tracking running averages of gradients and their squares — the default for deep learning. SGD with warm restarts and cyclical learning rates address the challenges of non-convex landscapes by periodically resetting.

Stochastic gradient descent (SGD) uses a random mini-batch of data to estimate $\nabla f$, trading variance for enormous speedups on large datasets. Remarkably, SGD often generalizes better than exact gradient descent — the noise acts as a regularizer and helps escape sharp minima. This is one of the deep mysteries (and joys) of deep learning.

Gradient descent is not guaranteed to find a global minimum in general — only a critical point. For non-convex losses (like neural network training), we can land at local minima, saddle points, or simply plateau. In practice, it is the primary algorithm for training every modern neural network — a testament to how well it navigates even these challenging landscapes when combined with good initialization, architectures, and stochasticity.