14 · Normal / Gaussian distribution
The bell curve that sits at the centre of statistics — two parameters (μ, σ), the 68-95-99.7 rule, and the limiting shape of countless sums.
The Normal is the central distribution of statistics — bell-shaped, two parameters (μ, σ), and the limiting form of countless sums (CLT).
Without this:
Without it, you can't speak the language of error bars, confidence intervals, regression residuals, or 99% of frequentist inference.
The Normal distribution (also called Gaussian) is the most important continuous distribution in statistics and machine learning. It is the canonical "bell curve" and arises naturally as the limiting shape of sums of independent random variables — the Central Limit Theorem (coming in the next chapter).
Probability Density Function (PDF): f(x) = (1 / (σ√(2π))) · exp(−(x − μ)² / (2σ²))
Two parameters completely determine the distribution:
- μ (mu) — the mean, controls the location (where the bell sits on the x-axis)
- σ (sigma) — the standard deviation, controls the scale (how wide or narrow the bell is)
We write X ~ N(μ, σ²), using σ² (variance) as the second parameter by convention.
Key properties:
- Symmetric around μ — the PDF is a mirror image left and right of μ
- Mean = Median = Mode = μ — all three coincide at the centre
- Tails extend to ±∞ — but fall off exponentially fast, so extreme values are very unlikely
- Closed under convolution: if X ~ N(μ₁, σ₁²) and Y ~ N(μ₂, σ₂²) are independent, then X + Y ~ N(μ₁+μ₂, σ₁²+σ₂²) — sums of normals are normal
The empirical (68-95-99.7) rule: Within 1σ of μ: ≈ 68.3% of the probability mass Within 2σ of μ: ≈ 95.4% of the probability mass Within 3σ of μ: ≈ 99.7% of the probability mass
This rule is the intuitive key to every "error bar" in science. If you know μ and σ, you immediately know the range where almost all observations will fall.
Python (in browser)
scipy.stats.norm(loc=μ, scale=σ) creates a Normal distribution object. The CDF tells us the probability of being below a threshold; subtracting two CDF values gives the mass in an interval. The 68-95-99.7 rule emerges from CDF arithmetic — no tables needed.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Three Normal PDFs on one figure. Blue: standard N(0,1). Yellow: N(0,2) — same center but twice as wide and correspondingly shorter (area must stay 1). Red: N(2,1) — same width as blue but shifted right by 2 units. μ translates, σ dilates. Open /tmp/normals.png to view.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
rng.normal(μ, σ, n) draws n samples from N(μ, σ²). With 10 000 draws the empirical mean, std, and the ±1/2/3σ fractions all converge tightly to their theoretical values — confirming the Normal model numerically.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The 68-95-99.7 rule is just 2·Φ(k) − 1 evaluated at k=1, 2, 3. The famous '1.96' is the z-value that makes the result exactly 0.95 — the workhorse of 95% confidence intervals everywhere in statistics.
For N(μ, σ), what proportion of mass lies within ±2σ of the mean?
Gaussian Distribution — the MML chapter that derives the Normal PDF from first principles, proves closure under linear transformation and convolution, and connects it to the multivariate Gaussian used in Bayesian ML.
- **PDF**: f(x) = (1/σ√(2π))·exp(−(x−μ)²/(2σ²)). **Mean = Median = Mode = μ**, **Var = σ²**. Symmetric, bell-shaped.
- **68-95-99.7 rule**: ±1σ contains 68%, ±2σ contains 95%, ±3σ contains 99.7% of the probability mass. The key to interpreting error bars.
- Sums of independent Normals are Normal (closed under convolution). The CLT means this distribution arises naturally from accumulated small effects.
Linear regression assumes Gaussian residuals; weight initialization in neural nets uses Normal (or its truncated variant); VAE latent priors are Normal; Bayesian regression conjugate priors are Normal.
If you remove it: Half the loss functions in supervised learning lose their justification — MSE-as-Gaussian-NLL collapses.