Probability Spaces

A probability space is a mathematical stage for reasoning about uncertainty. It consists of three ingredients: a sample space $\Omega$ (the set of all possible outcomes), a collection $\mathcal{F}$ of events (subsets of $\Omega$ we want to assign probabilities to), and a probability measure $P$ that assigns a number in $[0, 1]$ to each event. Together, $(\Omega, \mathcal{F}, P)$ axiomatize the notion of chance.

The probability measure $P$ satisfies three Kolmogorov axioms: (1) $P(A) \geq 0$ for every event $A$, (2) $P(\Omega) = 1$ — something must happen, and (3) countable additivity: for disjoint events $A_1, A_2, \dots$, $P(\bigcup_i A_i) = \sum_i P(A_i)$. From these three, you can derive everything: complements ($P(A^c) = 1 - P(A)$), unions (inclusion–exclusion), and monotonicity.

A random variable is a function $X : \Omega \to \mathbb{R}$ that assigns a number to each outcome. It lets us translate messy sample spaces into real numbers we can compute with. Crucially, we usually don't care about $\Omega$ itself — we care about the distribution of $X$, $P(X \in A)$ for sets $A \subseteq \mathbb{R}$. The distribution is the practical object.

Independence of events is defined by $P(A \cap B) = P(A) P(B)$ — the joint probability factorizes. Conceptually, knowing $A$ tells us nothing about $B$. Random variables $X, Y$ are independent if their joint distribution factors as the product of marginals: $p(x, y) = p(x) p(y)$. Independence is a very strong assumption that many ML models rely on (naive Bayes is the extreme example) — and whose violation has real consequences.

In ML, probability spaces formalize the data-generating process. We imagine each training example $(\mathbf{x}_i, y_i)$ as drawn independently from some underlying joint distribution $p(\mathbf{x}, y)$. Our goal is to estimate conditional probabilities $p(y \mid \mathbf{x})$ from a finite sample. Every learning algorithm — from linear regression to GPT — is implicitly or explicitly estimating some probability-space object.