26 · Bayes' theorem
Bayes' theorem turns 'P(test | disease)' into 'P(disease | test)' — the direction every diagnosis, classifier, and A/B test posterior actually needs.
Bayes' theorem turns 'P(test | disease)' into 'P(disease | test)' — the direction every diagnosis, classifier, and scientific update actually needs.
Without this:
Without it, you can't update beliefs with evidence — and almost every ML inference is a Bayesian update at some level.
Conditional probability gives us P(B | A) — the probability of B given that A happened. But what if we need P(A | B)? That's where Bayes' theorem comes in.
Derivation from the product rule:
The product rule of probability says P(A and B) = P(A) · P(B|A), and also P(A and B) = P(B) · P(A|B). Setting these equal and solving for P(A|B):
Bayes' theorem: P(A | B) = P(B | A) · P(A) / P(B)
Standard terminology:
- Prior P(A): Your belief about A before seeing evidence B. In ML, this is your regularization or initial model assumption.
- Likelihood P(B | A): How probable the evidence B is, if A were true. This is your model / generative process.
- Evidence (marginal) P(B): The total probability of B across all scenarios. It acts as a normalising constant.
- Posterior P(A | B): Your updated belief about A after seeing evidence B. This is what you actually want.
In short: posterior ∝ likelihood × prior.
The classic paradox — rare disease, accurate test:
A disease affects 1% of the population. A test for it has 99% sensitivity (P(+ | disease) = 0.99) and 95% specificity (P(− | no disease) = 0.95). You test positive. What's P(disease | +)?
Most people intuitively say ~99%. The correct answer is around 17% — because the disease is so rare that most positive tests are false positives. This counterintuitive result motivates every concept in this lesson.
Python (in browser)
Step-by-step Bayes calculation: with 1% prevalence and 99% sensitivity, a positive test only yields ~17% posterior probability of disease. The prior dominates when the base rate is low. A second positive test updates it further to ~78%.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Beta distribution as a conjugate prior for a coin's bias. Each flip is a Bayesian update: the posterior after the flip becomes the prior for the next. Two heads shift the distribution right (more likely biased toward H); one tail pulls it back toward 0.5.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The Bayes box translates probabilities into counts — making the low PPV (~17%) tangible: 495 false positives from the large healthy population overwhelm the 99 true positives.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Naive Bayes classifier formula and sklearn implementation pattern.
MML §34 derives the Sum and Product rules and Bayes' theorem from first principles. It also covers the Bayes filter (the recursive Bayesian update) which is the foundation of Kalman filters and particle filters used in robotics and time-series ML.
If the disease prevalence is 50% (not 1%), how would the posterior P(disease | positive test) change compared to the 1% prevalence case?
- **Bayes' theorem:** P(A|B) = P(B|A)·P(A) / P(B). Posterior ∝ likelihood × prior.
- With rare base rates, even a highly accurate test yields surprising amounts of false positives (the medical paradox). Always compute PPV, not just sensitivity.
- In ML: posterior = model output; likelihood = model; prior = regularisation. L2 = Gaussian prior on weights; L1 = Laplace prior.
Naive Bayes is one of the simplest classifiers and still strong for text (spam detection). Bayesian neural nets, MCMC-based regression, variational inference, and Bayesian optimization all bottom out at Bayes' rule.
If you remove it: You can't justify regularization as a prior-on-weights argument, which is the cleanest framing of why L2 prevents overfitting.