Sum Rule, Product Rule, Bayes' Theorem

Three rules underpin all of probability. The sum rule (or marginalization): $p(x) = \sum_y p(x, y)$ (or $\int p(x, y) \, dy$ in the continuous case). Given a joint distribution, you recover a marginal by summing/integrating out the other variable. This is how we go from 'full knowledge' to 'knowledge about $X$ alone'.

The product rule: $p(x, y) = p(x \mid y) p(y) = p(y \mid x) p(x)$. The joint probability factors into a conditional times a marginal in either direction. Chaining it gives us the chain rule of probability: $p(x_1, x_2, \dots, x_n) = p(x_1) p(x_2 \mid x_1) p(x_3 \mid x_1, x_2) \cdots$, essential for autoregressive models like language models.

Bayes' theorem is just the product rule rearranged: $p(x \mid y) = \frac{p(y \mid x) p(x)}{p(y)}$. It inverts a conditional: if we know how effects follow from causes ($p(y \mid x)$), we can reason about causes from observed effects ($p(x \mid y)$). The pieces have standard names: $p(x)$ is the prior, $p(y \mid x)$ is the likelihood, $p(x \mid y)$ is the posterior, and $p(y) = \sum_x p(y \mid x) p(x)$ is the evidence.

Bayes' rule is the engine of Bayesian inference: start with a prior belief about parameters $\theta$, observe data $\mathcal{D}$, and update to $p(\theta \mid \mathcal{D}) \propto p(\mathcal{D} \mid \theta) p(\theta)$. The evidence in the denominator is often intractable, which is why approximate methods — variational inference, MCMC, Laplace approximations — are so central to modern probabilistic ML.

Applications stretch from classic spam filters (Bayes' rule combining word likelihoods and class priors) to medical diagnosis (testing-rule problem: with a 99% accurate test and 1% prevalence, the posterior probability of disease given a positive test is only about 50%, a frequently botched intuition) to modern Bayesian neural networks and diffusion models. Fluency with these three rules is the baseline for all probabilistic reasoning.