Gaussian Distribution

The Gaussian (or Normal) distribution is the most important distribution in statistics and ML. A univariate Gaussian has density $p(x) = \frac{1}{\sqrt{2\pi \sigma^2}} \exp\left(-\frac{(x - \mu)^2}{2\sigma^2}\right)$, parameterized by mean $\mu$ and variance $\sigma^2$. Its classic bell shape: symmetric around $\mu$, inflection points at $\mu \pm \sigma$, and roughly 99.7% of mass within $\mu \pm 3\sigma$.

The multivariate Gaussian $\mathcal{N}(\boldsymbol\mu, \Sigma)$ on $\mathbb{R}^d$ has density $p(\mathbf{x}) = \frac{1}{\sqrt{(2\pi)^d \det\Sigma}} \exp\left(-\tfrac{1}{2}(\mathbf{x} - \boldsymbol\mu)^\top \Sigma^{-1}(\mathbf{x} - \boldsymbol\mu)\right)$. The level sets are ellipsoids aligned with the eigenvectors of $\Sigma$ — the same eigenvectors PCA recovers. The mean is the center; the covariance describes the shape and orientation.

Gaussians are closed under several operations: sums of independent Gaussians are Gaussian, affine transformations $A\mathbf{x} + \mathbf{b}$ of Gaussian vectors are Gaussian, marginals are Gaussian, and conditionals are Gaussian. The conditional formulas $p(\mathbf{x}_1 \mid \mathbf{x}_2)$ have closed forms — the workhorses of Gaussian processes and Kalman filters.

Why are Gaussians everywhere? Three reasons. The central limit theorem says sums of many independent variables become approximately Gaussian, so noise is often modeled as Gaussian. The maximum entropy principle says the Gaussian is the 'most uncertain' distribution given fixed mean and variance, making it the least-committal choice. And practically, the Gaussian's closed-form algebra means tractable inference.

In ML, Gaussians power linear regression's noise model ($y = \mathbf{w}^\top \mathbf{x} + \epsilon$, $\epsilon \sim \mathcal{N}$), variational autoencoders (Gaussian latents and decoder posteriors), diffusion models (Gaussian transition kernels), and Gaussian processes (distributions over functions). Understanding Gaussian algebra — conditionals, marginals, and precision matrices — is arguably the most practically useful probability skill in ML.

Worked example — the 68/95/99.7 rule: For standard normal $Z \sim \mathcal{N}(0,1)$, $P(-1 \le Z \le 1) \approx 0.6827$, $P(-2 \le Z \le 2) \approx 0.9545$, and $P(-3 \le Z \le 3) \approx 0.9973$. These are the three most useful numerical facts about any Gaussian — outliers beyond $3\sigma$ occur about 0.3% of the time.

Worked example — 2D Gaussian contour: Take $\boldsymbol\mu = (0,0)$ and $\Sigma = \begin{pmatrix}4 & 0 \\ 0 & 1\end{pmatrix}$. The level set $(\mathbf{x} - \boldsymbol\mu)^\top \Sigma^{-1} (\mathbf{x} - \boldsymbol\mu) = 1$ becomes $x_1^2/4 + x_2^2 = 1$ — an ellipse with semi-axes 2 and 1 along the coordinate axes. The eigenvectors of $\Sigma$ give the principal directions; eigenvalues $(4, 1)$ give the squared axis lengths.

Worked example — affine transform stays Gaussian: If $X \sim \mathcal{N}(2, 9)$ and $Y = 3X - 1$, then $Y \sim \mathcal{N}(3\cdot 2 - 1,\; 3^2 \cdot 9) = \mathcal{N}(5, 81)$. So $Y$ has mean 5 and standard deviation 9 — closed form, no integral needed.