8 · Sum rule & product rule
Two rules — sum and product — generate every probability you'll ever compute, from dice rolls to Bayes' theorem.
Two rules — sum and product — generate every probability you'll ever compute.
Without this:
Without them, you can't combine evidence, condition on context, or write a likelihood.
Probability has three axioms that everything else is built on:
- Non-negativity: P(A) ≥ 0 for any event A.
- Normalization: P(sample space) = 1 — something always happens.
- Additivity: if A and B are mutually exclusive (A ∩ B = ∅), then P(A ∪ B) = P(A) + P(B).
From these three seeds, two rules do almost all the work:
Sum rule (addition rule): P(A ∪ B) = P(A) + P(B) − P(A ∩ B)
The subtraction corrects for double-counting the overlap. When A and B are mutually exclusive the overlap is empty (P(A ∩ B) = 0) and the formula collapses to axiom 3.
Product rule (multiplication rule): P(A ∩ B) = P(A) · P(B | A)
It follows directly from the definition of conditional probability: P(B | A) = P(A ∩ B) / P(A).
Independence: B is independent of A if and only if knowing A happened tells you nothing about B — formally P(B | A) = P(B). Substituting into the product rule gives the simpler form P(A ∩ B) = P(A) · P(B).
These six lines are the complete grammar of probability.
Python (in browser)
With 100 000 simulated rolls the empirical probability of a sum-7 converges to the theoretical 6/36 ≈ 0.1667. The law of large numbers at work.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The sum rule formula P(A∪B) = P(A)+P(B)−P(A∩B) gives 11/36. The naive P(A)+P(B) = 12/36 overcounts because the outcome (6,6) belongs to both events simultaneously.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The empirical joint probability P(die1=6 AND die2=6) matches P(die1=6) · P(die2=6) = 1/36 within sampling noise, confirming independence. Each die knows nothing about the other.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
A two-step probability tree encodes the product rule at each branch. Reading leaves bottom-up with the sum rule gives marginal probabilities. Bayes' theorem (next lesson) inverts the tree.
Two cards are drawn WITHOUT replacement from a shuffled 52-card deck. Are the events 'first card is a heart' (A) and 'second card is a heart' (B) independent?
Sum, Product, and Bayes' Rules — the MML chapter that derives both rules formally from the axioms, proves the chain rule for n events, and shows how Bayes' theorem is a one-line consequence of the product rule.
- **Sum rule**: P(A ∪ B) = P(A) + P(B) − P(A ∩ B). The subtraction prevents double-counting the overlap. For mutually exclusive events the overlap is zero.
- **Product rule**: P(A ∩ B) = P(A) · P(B | A). For **independent** events P(B | A) = P(B), so it simplifies to P(A) · P(B).
- **Conditional probability** P(B | A) = P(A ∩ B) / P(A) is the building block of all ML inference — computing P(label | features) is the universal supervised learning task.
Bayes' theorem (next lesson) is a one-line algebraic rearrangement of the product rule. Naive Bayes classifiers assume features are conditionally independent — the product rule is the engine.
If you remove it: You can't even write down a likelihood P(data | parameters), which is the universal supervised learning objective.