22 · What the p-value means (and doesn't)
p = P(data this extreme | H₀ true) — not P(H₀ | data). Know this distinction to avoid one of science's most common errors.
The p-value is P(data this extreme | H₀ true) — NOT P(H₀ true | data).
Without this:
Without this distinction, you'll join the long list of researchers who misinterpret p < 0.05 as 'the null is 95% likely false'.
The p-value is the most misunderstood number in science. Here is the precise definition, followed by the most common misinterpretations to avoid.
Precise definition: p-value = P(observing a test statistic this extreme, or more extreme | H₀ is true)
This is a conditional probability. The condition is "H₀ is true." The p-value lives entirely in the world where H₀ holds, and asks: "Is our observed data compatible with that world?"
Common misinterpretations (AVOID ALL OF THESE):
-
"p = 0.03 means there is only a 3% chance H₀ is true." WRONG. To get P(H₀ | data) you need Bayes' theorem and a prior on H₀. The p-value gives P(data | H₀), not P(H₀ | data).
-
"p = 0.03 means there is a 97% chance the result will replicate." WRONG. The replication probability depends on sample size, effect size, and study design — not directly on p.
-
"p < 0.05 means the effect is practically important." WRONG. With enough data (n = 1,000,000), even a 0.0001% difference will give p < 0.05. Statistical significance ≠ practical significance.
-
"p > 0.05 means H₀ is true." WRONG. It means we couldn't rule out H₀ with this sample size. "Fail to reject" ≠ "accept."
p-hacking and multiple testing: If you test 100 independent hypotheses and none is true, you still expect 5 to have p < 0.05 by chance (5% false-positive rate × 100 tests = 5 expected false positives). Running many tests on the same data without correction inflates the false-positive rate — this is p-hacking or the multiple testing problem.
Corrections:
- Bonferroni: require p < α/m per test, where m is the number of tests. Controls the family-wise error rate (FWER) — the probability of ANY false positive.
- Benjamini-Hochberg (BH/FDR): allows up to a fraction q of all discoveries to be false positives (false discovery rate). Less conservative than Bonferroni; standard in genomics and ML feature selection.
Python (in browser)
Under H₀ (no true effect), p-values are uniformly distributed on [0, 1]. About 5% of experiments will produce p < 0.05 purely by chance. This is the mathematical reason why running many tests without correction leads to false positives.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
With 1000 features, none related to the label, roughly 50 will have p < 0.05 by chance (5% × 1000 = 50). Selecting these features would include ~50 pure-noise predictors in your model — a classic cause of train/test performance gaps.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Bonferroni correction requires p < 0.05/1000 = 5×10⁻⁵ per test. This nearly eliminates false positives among the null features. The trade-off is reduced power to detect real effects — but in feature selection, false positives are especially harmful.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The four forbidden interpretations: (1) it's not P(H₀ | data); (2) it's not the replication probability; (3) statistical significance ≠ practical importance; (4) p > 0.05 doesn't confirm H₀. The correct reading is purely about compatibility with H₀, nothing else.
If p = 0.03, the probability that H₀ is true is…
- **p-value = P(data this extreme | H₀ true)** — NOT P(H₀ | data). The latter requires Bayes.
- Under H₀, p-values are Uniform[0, 1] — exactly 5% will be < 0.05 by chance per test.
- Multiple testing: m tests → expect m×α false positives. Correct with Bonferroni (α/m) or Benjamini-Hochberg FDR.
Multiple testing is everywhere in feature selection (1000 features × 1 label = 1000 tests). A/B test platforms must adjust for the number of metrics tested. Hyperparameter tuning across many configs is implicit multiple testing.
If you remove it: You'd ship 'significant' features that are pure noise — the classic 'why does my model fail on new data?' story.