5 · Variance, std dev & why n−1
Spread matters as much as centre — and one correction factor explains why every stats library has a 'ddof' argument.
Variance is the expected squared distance from the mean — and dividing by n−1 instead of n is not a typo; it removes a systematic bias.
Without this:
Without variance and std dev, you can't z-score features, understand PCA, or interpret model loss curves.
Knowing the centre of a distribution is only half the picture. Two datasets can share the same mean yet look completely different:
- Salaries of {50k, 50k, 50k, 50k} — perfectly flat.
- Salaries of {10k, 30k, 70k, 90k} — huge spread.
Dispersion measures characterize the spread of a distribution:
- Variance σ² = E[(X − μ)²]: average squared deviation from the mean. Squaring has two effects — it makes all deviations positive, and it amplifies large deviations more than small ones.
- Standard deviation σ = √(Variance): back to the original units (dollars, milliseconds, degrees…), making it directly comparable to the data.
- Range = max − min: the simplest measure; catastrophically sensitive to a single outlier.
- Interquartile Range (IQR) = Q3 − Q1: the span of the middle 50 % of the data. Robust to outliers because the tails are irrelevant.
- Mean Absolute Deviation (MAD) = E[|X − μ|]: averages absolute (not squared) deviations. More robust than variance but less mathematically convenient.
The key tradeoff: variance and std dev are sensitive to outliers; IQR and MAD are robust.
Python (in browser)
ddof=0 divides by n (population formula). ddof=1 divides by n−1 (sample formula, Bessel's correction). The sample variance is always larger by a factor of n/(n−1).
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Why n−1? Bessel's correction explained.
When you estimate variance from a sample, you first compute the sample mean x̄ from that same data. Here's the subtle problem: the sample mean x̄ is not the true population mean μ — it's the mean that minimizes the sum of squared deviations from the data you have. That means the deviations (xᵢ − x̄) are systematically smaller than the deviations from the true μ. Dividing by n therefore underestimates the true variance.
Dividing by n−1 corrects this bias. Intuitively: once you've fixed x̄, only n−1 of the deviations are "free" — the last one is determined by the constraint that all deviations must sum to zero.
The formal statement: E[S²] = σ² only when S² = Σ(xᵢ − x̄)² / (n−1). This is what "unbiased estimator" means.
Python (in browser)
With 10,000 simulations of samples of size 5, the n-divisor average consistently lands below 1.0 (true variance), while the n−1-divisor average lands near 1.0. This is what 'unbiased' means in practice.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Std dev triples under 5% contamination; IQR is unmoved. For datasets with potential outliers (sensor errors, annotation mistakes), IQR is the safer spread measure.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
If σ = 15 for a normal distribution, approximately what proportion of values fall within ±σ of the mean (i.e., between μ−15 and μ+15)?
Summary Statistics and Independence — the MML chapter that formalizes variance as E[(X−μ)²], covariance, and the independence of random variables. Read it for the derivation of E[S²] = σ² under Bessel's correction.
- **Variance** = average squared deviation from the mean. **Std dev** = √variance, in the same units as the data. Both are sensitive to outliers.
- Dividing by **n−1** (Bessel's correction) gives an *unbiased* sample variance. numpy uses ddof=0 by default; pandas uses ddof=1. Always be explicit.
- **IQR** (Q3 − Q1) is the robust alternative to std dev — it ignores the tails entirely and is unaffected by outliers.
Standard deviation is the scaling factor in z-score normalization `(x − μ) / σ`. Variance is what PCA maximizes in each principal component. Loss-curve uncertainty bands are typically ±1 std dev across bootstrap or cross-validation runs.
If you remove it: Feature scaling without σ is impossible — you can't compute z-scores or normalize gradients. PCA has no defined objective without the notion of explained variance.