Discrete and Continuous Probabilities

Random variables come in two flavors. A discrete random variable takes values in a finite or countable set (like $\{1, 2, 3, 4, 5, 6\}$ or $\mathbb{N}$) and is described by a probability mass function (PMF): $p(x) = P(X = x)$. Mass functions satisfy $p(x) \geq 0$ and $\sum_x p(x) = 1$ — probability is distributed across discrete points.

A continuous random variable takes values in an uncountable set like $\mathbb{R}$, and is described by a probability density function (PDF) $p(x)$ with $p(x) \geq 0$ and $\int p(x) \, dx = 1$. The probability of any single point is zero; meaningful probabilities come from intervals: $P(a \leq X \leq b) = \int_a^b p(x) \, dx$.

Densities are *not* probabilities — they can be greater than 1 (a narrow tall density integrates to 1 over a small region). This surprises people. The right mental picture: a density is a rate, probability-per-unit-of-$x$, and what matters is the area under it over an interval. Units matter too: if $x$ is in meters, $p(x)$ has units of 1/meter.

Both types share a cumulative distribution function (CDF): $F(x) = P(X \leq x)$. For discrete variables, the CDF is a staircase; for continuous, it's a smooth increasing function. The CDF is universally useful because it always exists, always returns true probabilities, and maps cleanly to quantile-based computations (sampling via inverse CDF, confidence intervals).

ML mixes both constantly. Classification outputs a discrete distribution over labels via softmax. Regression predicts a continuous $y$ given $\mathbf{x}$. Tokens in a language model are discrete but attention scores are continuous. Mastery of both paradigms — and especially how to think about *mixed* or *structured* distributions like those in energy-based models — is essential for building and debugging modern probabilistic systems.

Worked example — PMF of Binomial(3, 0.5): For $X \sim \text{Binomial}(3, 0.5)$, $p(k) = \binom{3}{k}(0.5)^3$ gives $p(0) = 1/8$, $p(1) = 3/8$, $p(2) = 3/8$, $p(3) = 1/8$. Total: $1/8 + 3/8 + 3/8 + 1/8 = 1$. These are exactly the probabilities of 0, 1, 2, 3 heads in three fair flips.

ML connection — softmax as a PMF: Given logits $(1, 2, 3)$, softmax outputs $p_i = e^{z_i}/\sum_j e^{z_j}$ giving roughly $(0.09, 0.24, 0.67)$, summing to $1$ and all non-negative. This is precisely a PMF over three discrete classes — the reason softmax is the canonical classification output.