25 · Type I & II errors and power
Every test can fail in exactly two ways: reject when you shouldn't (Type I, α) or miss a real effect (Type II, β). Power = 1 − β is what you maximise by choosing n.
Two ways a test can be wrong: reject a true null (Type I, α) or fail to reject a false null (Type II, β). Power = 1 − β.
Without this:
Without the error framework, you can't size experiments — you'll either over-test or miss real effects.
A hypothesis test is a decision rule: it says "reject H₀" or "don't reject H₀" based on data. Like any decision, it can be wrong in two different ways:
| | H₀ is TRUE | H₀ is FALSE | |---|---|---| | Reject H₀ | Type I error (false positive) — probability = α | Correct! — probability = power = 1 − β | | Don't reject H₀ | Correct! — probability = 1 − α | Type II error (false negative) — probability = β |
Key definitions:
- α (significance level): The probability of a Type I error — rejecting H₀ when it is actually true. You choose α before the test; typical values are 0.05 or 0.01.
- β: The probability of a Type II error — failing to reject H₀ when H₁ is actually true.
- Power = 1 − β: The probability of correctly detecting a real effect. Power ≥ 0.80 (80%) is the standard target.
What controls these?
- Making α smaller (more conservative) → β increases (less power), for fixed n.
- Increasing n → β decreases (more power) without changing α.
- Larger effect size Δ → β decreases (easier to detect bigger differences).
- More variable data (larger σ) → β increases (harder to detect effects in noise).
The ROC connection: In a binary classifier, the FPR (false positive rate) = α and the TPR (true positive rate = recall) = power. The ROC curve is literally a plot of power vs α as you vary the decision threshold. AUC summarises the tradeoff across all thresholds.
Python (in browser)
The defining figure: H₀ (blue) and H₁ (green) as overlapping bell curves. The red area is Type I error — rejecting a true H₀. The yellow area is Type II error — failing to detect a true H₁. Moving the critical value left reduces Type I but inflates Type II, and vice versa. The only way to shrink both simultaneously is to increase n.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Running 100,000 t-tests verifies both error rates empirically: under H₀ we see ~5% false positives (by design), and under H₁ with a medium effect (d=0.5, n=30) we detect ~70% of true effects.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Power climbs from ~20% at n=10 to >99% at n=300. For a medium effect (d=0.5) at the standard α=0.05, you need ~64 participants per group to reach 80% power. This is the sample-size calculation every experiment design starts with.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The practical sample-size calculation workflow using statsmodels.
A study reports power = 0.80. What is β (the Type II error rate)?
- **Type I (α):** reject a true H₀ — false positive. **Type II (β):** fail to reject a false H₀ — false negative. **Power = 1 − β.**
- Smaller α → larger β for fixed n. More n reduces both simultaneously. Larger effect size → easier to detect (smaller β).
- In a classifier: FPR = α, TPR = power. The ROC curve plots power vs α across all thresholds. AUC summarises the tradeoff.
ROC/AUC frameworks ARE this — TPR (power) vs FPR (α). Choosing a classification threshold is a Type I/II tradeoff. Multiple testing inflates α; FDR controls it.
If you remove it: You can't choose a classifier threshold defensibly without explicit error-rate thinking.