21 · Hypothesis testing mechanism
State a null, pick a test statistic, compute the null distribution, measure how surprising your data is — the six-step framework that powers every A/B test.
State a null, pick a test statistic, compute its distribution under the null, see how 'surprising' your observed data is — that's hypothesis testing.
Without this:
Without the framework, A/B testing is guesswork and 'statistical significance' is meaningless jargon.
Hypothesis testing is a formal framework for deciding whether an observed pattern in data is real or just noise. The core question: "Is this observation compatible with random chance, or should we update our beliefs?"
The six-step procedure:
-
State H₀ (null hypothesis) and H₁ (alternative). H₀ is the "boring" default — no effect, no difference. H₁ is what you're trying to prove. Example: H₀: μ = 100 (IQ scores are average); H₁: μ > 100 (scores are above average).
-
Choose the significance level α. Typically α = 0.05 (5%). This is the maximum false-positive rate you're willing to tolerate — the probability of rejecting H₀ when it is actually true.
-
Compute the test statistic from the data. The test statistic is a number that measures how far the observed data is from what H₀ predicts. For testing a mean with known σ, the z-statistic is: z = (x̄ − μ₀) / (σ/√n).
-
Derive the distribution of that statistic UNDER H₀. If H₀ is true, z ~ N(0, 1) (by the CLT). This gives us the "null distribution" — the reference we compare against.
-
Compute the p-value. The p-value is the probability of observing a test statistic at least as extreme as the one we got, assuming H₀ is true. For a one-tailed test: p = P(Z ≥ z_observed | H₀). For two-tailed: p = 2 · P(Z ≥ |z_observed| | H₀).
-
Make a decision. If p < α, reject H₀ ("statistically significant"). If p ≥ α, fail to reject H₀ (we can't rule out chance — this is not the same as "H₀ is true").
One-tailed vs two-tailed:
- One-tailed when the alternative is directional: H₁: μ > μ₀ or H₁: μ < μ₀.
- Two-tailed when any deviation matters: H₁: μ ≠ μ₀. Two-tailed is the safer default — use one-tailed only when theory strongly predicts the direction before seeing data.
Python (in browser)
z = (105 − 100) / (15/√30) ≈ 1.826. The one-tailed p-value ≈ 0.034 < 0.05 → reject H₀. There is statistically significant evidence that the sample comes from a population with mean above 100.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The Monte Carlo simulation directly illustrates the p-value definition: under H₀, roughly 3-4% of experiments produce a z-statistic as large as the observed one. This empirical fraction agrees with the theoretical Normal-CDF p-value.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
One-tailed tests have more power to detect effects in the predicted direction, but require a pre-committed directional hypothesis. Two-tailed tests (p × 2) are the conservative default and protect against detecting spurious 'effects' in the wrong direction.
What does 'p < 0.05' mean?
- Six steps: (1) H₀ & H₁, (2) α, (3) test statistic, (4) null distribution, (5) p-value, (6) reject if p < α.
- **p-value = P(data this extreme | H₀).** It is NOT the probability H₀ is true.
- Default to two-tailed unless a directional hypothesis was pre-committed before data collection.
A/B testing IS hypothesis testing. Comparing two models' accuracies requires a paired test. Feature selection often uses per-feature tests against the null 'no effect'.
If you remove it: You can't formally answer 'is model B better than model A on the test set, or did it get lucky?'