Linearization and Taylor Series

A Taylor series approximates a smooth function by a polynomial. Near a point $a$, the $k$-th order approximation is $f(x) \approx f(a) + f'(a)(x - a) + \frac{f''(a)}{2!}(x-a)^2 + \dots + \frac{f^{(k)}(a)}{k!}(x - a)^k$. The first-order approximation is the tangent line (linearization); the second-order version is a parabola tangent to the function at $a$.

For multivariable $f : \mathbb{R}^n \to \mathbb{R}$, the Taylor expansion becomes $f(\mathbf{x}) \approx f(\mathbf{a}) + \nabla f(\mathbf{a})^\top (\mathbf{x} - \mathbf{a}) + \frac{1}{2} (\mathbf{x} - \mathbf{a})^\top H_f(\mathbf{a}) (\mathbf{x} - \mathbf{a})$. Linearization uses only the gradient; quadratic approximation adds the Hessian term. This is the bedrock of almost all continuous optimization theory.

Approximation quality depends on how far you are from the base point and how non-polynomial $f$ is. The remainder term $R_k(x)$ typically decays as $|x - a|^{k+1}$ for smooth functions, so small neighborhoods admit very accurate low-order approximations. For smooth functions analytic in a neighborhood, the series converges exactly to $f$ — but not every smooth function has this property (a famous counter-example is $e^{-1/x^2}$, smooth but not analytic at 0).

Linearization is a recurring idea in ML. Gradient descent uses the 1st-order Taylor approximation to propose a step. Newton's method uses the 2nd-order approximation and jumps directly to the minimum of the local quadratic model. Natural gradient uses a Riemannian correction based on the Fisher metric. Trust region methods use quadratic models but constrain step size to where the model is trustworthy.

Beyond optimization, Taylor series power Laplace approximations (approximate a posterior by its 2nd-order expansion around the mode), extended Kalman filters (linearize nonlinear dynamics at each step), and numerical solvers for ODEs and PDEs. Even approximating activation functions by low-degree polynomials for homomorphic-encryption-friendly inference is a Taylor-series maneuver. It is the universal 'zoom in close and pretend things are simple' tool.

Worked example — Taylor series of $\cos(x)$ around 0 up to order 4: $\cos(x) \approx 1 - \frac{x^2}{2} + \frac{x^4}{24}$. At $x = 0.1$ this gives $1 - 0.005 + 0.0000042 \approx 0.9950042$, while the true value is $\cos(0.1) \approx 0.9950042$ — indistinguishable to six decimals. The approximation error shrinks like $|x|^6/720$, which is about $10^{-9}$ at $x = 0.1$.