29 · Variance partitioning & ANOVA deep dive (capstone)
ANOVA's secret is that total variation = between-group + within-group variation. That identity powers R², eta-squared, and ultimately the whole of regression.
ANOVA's secret is variance partitioning: total variation in the data equals between-group variation plus within-group variation.
Without this:
Without the partition view, you can't reason about R² (the proportion of variance explained) or compute effect sizes like eta-squared.
In the previous lesson you learned how to run ANOVA with scipy.stats.f_oneway. Now we open the hood and see why the F-test works — through variance partitioning.
The fundamental ANOVA identity:
SST = SSB + SSW
where:
- SST (total sum of squares) = Σᵢ (xᵢ − grand_mean)² — total variation in all the data.
- SSB (between-group SS) = Σⱼ nⱼ (x̄ⱼ − grand_mean)² — variation due to group membership.
- SSW (within-group SS) = Σⱼ Σᵢ (xᵢⱼ − x̄ⱼ)² — variation within each group (pure noise).
This always holds exactly — it is an algebraic identity, not an approximation.
From these sums of squares we get mean squares (variance estimates):
| | Formula | df | |--|---------|-----| | MSB (between) | SSB / (k − 1) | k − 1 | | MSW (within) | SSW / (N − k) | N − k | | F | MSB / MSW | — |
Eta-squared (η²):
η² = SSB / SST
This is the proportion of total variance explained by group membership — identical to R² in linear regression. A model with η² = 0.30 explains 30% of the variance in the outcome through group differences alone.
The connection to regression:
One-way ANOVA is literally linear regression with a one-hot encoded categorical predictor. If you create dummy variables for k groups (dropping one as reference) and run statsmodels.ols, the F-statistic and p-value printed at the bottom of the summary are the same as f_oneway. The R² in the regression output equals η².
This means R² — the quantity every data scientist reads first in a regression output — is just ANOVA's variance partition applied to a continuous predictor.
Python (in browser)
Manual variance partitioning: compute SST, SSB, SSW by hand, verify SST = SSB + SSW exactly, then derive MSB, MSW, F, and η². This is what f_oneway computes internally.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Checking ANOVA assumptions: Levene's test for equal variances (if p < 0.05, switch to Welch's ANOVA) and Shapiro-Wilk for normality within each group.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
When Levene's test signals unequal variances, Welch's ANOVA (scipy.stats.alexandergovern) is the robust alternative. It adjusts degrees of freedom to account for heteroscedasticity.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
ANOVA = linear regression with dummy-coded categorical predictors. The F-statistic and R² (= η²) are identical in both formulations. This makes ANOVA the conceptual ancestor of the entire regression family.
In a one-way ANOVA, SSB = 30 and SSW = 70. What is η² (eta-squared)?
The variance partitioning identity SST = SSB + SSW and its matrix algebra form underpin everything from linear regression to PCA. Review MML chapter 35 for the full treatment.
- **Decomposition:** SST = SSB + SSW (total sum of squares = between + within). This identity always holds exactly.
- **Mean squares:** MSB = SSB/(k−1), MSW = SSW/(N−k). F = MSB / MSW.
- **Eta-squared:** η² = SSB / SST — the proportion of total variance explained by group membership. Identical to R² in a one-way ANOVA regression.
- **Connection to regression:** ANOVA = linear regression with one-hot encoded categorical predictor. The F-test in both outputs is the same test.
R² in linear regression, AUC variance decomposition, ablation studies comparing model variants. ANOVA is the conceptual ancestor of regression, generalized linear models, and analysis of deviance.
If you remove it: You can't reason about 'how much of the variance does this feature explain?' — the foundational question of regression diagnostics.