28 · ANOVA: comparing 3+ group means
When you have 3 or more groups and a continuous outcome, one F-test replaces all pairwise t-tests — keeping your Type I error under control.
ANOVA generalizes the t-test to 3+ groups — by analyzing variance ratios, not means directly.
Without this:
Without it, you'd run pairwise t-tests for every pair (inflating Type I error) instead of one unified test.
You've already seen the t-test for comparing two group means. But what if you have three experimental conditions — drug A, drug B, and placebo? Running three separate t-tests (A vs. B, A vs. placebo, B vs. placebo) creates a problem: each test has a 5% chance of a false positive at α = 0.05. With 3 tests you'd expect at least one spurious significant result about 14% of the time even when all groups are identical.
Analysis of Variance (ANOVA) solves this with a single test.
The core insight — variance ratios, not mean differences:
Instead of comparing means directly, ANOVA asks: is the variation between groups larger than what we'd expect from random within-group variation?
F = Variance BETWEEN groups / Variance WITHIN groups
- Between-group variance (MSB): how much the group means vary from the grand mean — inflated by real treatment effects and by random noise.
- Within-group variance (MSW): the average variance of observations around their own group mean — driven only by random noise.
Under H₀ (all group means equal), both numerator and denominator estimate the same population variance, so F ≈ 1. Under H₁ (at least one mean differs), the numerator grows while the denominator stays the same, so F > 1.
Assumptions:
- Observations are independent (no repeated measures).
- Within each group, data is approximately Normal.
- Groups have equal variances (homoscedasticity — Bartlett's or Levene's test can check this).
When assumptions fail: Welch's ANOVA (relaxes equal variance) or Kruskal-Wallis (fully non-parametric).
Python (in browser)
One-way ANOVA on three simulated drug groups. With σ = 15 and true mean differences of 5 and 1 point, the test may or may not reject depending on the random seed — illustrating that ANOVA's power depends on effect size relative to within-group noise.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Boxplots with jittered individual data points for all three groups. The dashed line marks the grand mean — visually you can see Drug B's box is a bit higher, but the overlap is substantial, which is why the formal test matters.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Simulation of the family-wise error rate: 5 groups all sampled from the same distribution, yet 10 pairwise t-tests produce at least one spurious 'significant' result in roughly 40% of experiments. ANOVA collapses this to a single 5% test.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
ANOVA assumptions, one-way vs two-way, and the F-statistic decomposition — the conceptual scaffolding before we compute it by hand in the next lesson.
If the F-statistic in a one-way ANOVA equals 1, what does that indicate?
- **F-statistic:** F = (variance BETWEEN groups) / (variance WITHIN groups). Under H₀ (all means equal), F ≈ 1.
- ANOVA's H₀: all group means are equal. H₁: at least one group mean differs. It does NOT tell you which pair differs.
- **Post-hoc problem:** after ANOVA rejects, use Tukey HSD or pairwise t-tests with Bonferroni correction to find which pairs differ.
- **Assumptions:** independence of observations, approximate normality within each group, and equal variances (homoscedasticity).
Comparing 3+ model architectures' performance (each tested on the same dataset); A/B/C/D testing with 4 variants; feature-importance for categorical variables with 3+ levels.
If you remove it: You'd run pairwise tests and inflate false-discovery rates badly.