Differentiation of Univariate Functions

The derivative $f'(x)$ of a function $f : \mathbb{R} \to \mathbb{R}$ measures its instantaneous rate of change at $x$. Formally, $f'(x) = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h}$. Geometrically, it is the slope of the tangent line to the graph of $f$ at the point $(x, f(x))$.

A function is differentiable at $x$ if this limit exists. Differentiability implies continuity, but not conversely — the absolute value $|x|$ is continuous everywhere but not differentiable at $0$, where the graph has a sharp corner. More broadly, functions with kinks, cusps, or vertical tangents fail to be differentiable at those points.

A few derivative rules do most of the work: $\frac{d}{dx}x^n = nx^{n-1}$ (power rule), $\frac{d}{dx} e^x = e^x$, $\frac{d}{dx} \ln x = 1/x$, $\frac{d}{dx} \sin x = \cos x$, $\frac{d}{dx} \cos x = -\sin x$. Combined with linearity $(af + bg)' = af' + bg'$, the product rule $(fg)' = f'g + fg'$, and the chain rule $(f \circ g)'(x) = f'(g(x))\cdot g'(x)$, you can differentiate almost any elementary function.

Derivatives are how we find extrema: at a local minimum or maximum of a smooth function, $f'(x) = 0$. The second derivative $f''(x)$ tells us the concavity — positive means a local min (curving up), negative means a local max (curving down). This is the foundation of optimization.

In ML, univariate differentiation is everywhere: computing the derivative of a loss with respect to one scalar parameter, tuning a learning rate by analyzing how a scalar-valued update behaves, understanding activation functions like $\sigma(x)$ or ReLU via their (sub)derivatives. Before we can take gradients of complex neural networks, we need rock-solid comfort with 1D derivatives as the atomic building block.

ML connection — derivative of the sigmoid at zero: Since $\sigma(x) = 1/(1+e^{-x})$ and $\sigma'(x) = \sigma(x)(1-\sigma(x))$, at $x=0$ we get $\sigma(0) = 0.5$ and $\sigma'(0) = 0.5\cdot 0.5 = 0.25$. Far from zero, say at $x = 5$, $\sigma(5) \approx 0.993$ so $\sigma'(5) \approx 0.007$ — this tiny slope is exactly the vanishing-gradient pain that motivated ReLU.