11 · Bernoulli: single yes/no trial
The simplest distribution in the book — a single trial with two outcomes and one parameter. Every binary classifier is built on this.
The Bernoulli is the simplest distribution — a single trial with two outcomes, parameter p.
Without this:
Without it, you can't model a coin flip, a binary classification label, or a click/no-click event.
A Bernoulli random variable is the simplest possible distribution: a single trial that has exactly two outcomes — success (1) with probability p, or failure (0) with probability 1 − p.
Probability Mass Function (PMF):
- P(X = 1) = p
- P(X = 0) = 1 − p
- Written compactly: P(X = k) = p^k · (1 − p)^(1−k) for k ∈ {0, 1}
Mean and variance:
- E[X] = p (the average of 0s and 1s from many trials equals the success probability)
- Var(X) = p(1 − p)
Maximum variance at p = 0.5: The variance function p(1 − p) is a downward-opening parabola with maximum at p = 0.5, where Var(X) = 0.25. This is the point of maximum uncertainty — when a coin is perfectly fair, you know the least about any single toss. At p = 0 or p = 1 the outcome is certain, so variance collapses to 0.
Connection to binary classification: Every supervised binary classification example carries a label y ∈ {0, 1}. That label is a draw from a Bernoulli(p) distribution, where p = P(positive class | features). The model's job is to estimate p given the features. This is why logistic regression outputs a value in (0, 1) — it is estimating the Bernoulli parameter.
Log-likelihood and binary cross-entropy: The likelihood of observing label y given model probability p is: L(p ; y) = p^y · (1 − p)^(1−y)
Taking the log: log L = y·log(p) + (1−y)·log(1−p)
Negating to turn maximization into minimization: −log L = −y·log(p) − (1−y)·log(1−p)
This is exactly the binary cross-entropy loss. The loss function in logistic regression IS the Bernoulli negative log-likelihood.
Python (in browser)
scipy.stats.bernoulli(p) gives you a distribution object. .mean() returns p, .var() returns p(1−p), and .rvs(n) draws n independent samples. With only 10 samples the empirical mean is noisy — that's expected.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The variance curve Var(X) = p(1−p) peaks at p=0.5 (Var=0.25) and falls to zero at both extremes. This parabola is the geometric picture of uncertainty: a fair coin is the hardest to predict; a completely biased one is trivial. Open /tmp/bern_var.png to view.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
With 10 000 draws the empirical mean converges to p=0.3 (≈ 0.21 variance). rng.binomial(1, p, size=n) is the canonical way to generate Bernoulli samples with NumPy — it is simply the Binomial with n=1.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Binary cross-entropy is not arbitrary — it falls directly out of the Bernoulli negative log-likelihood. When you call model.compile(loss='binary_crossentropy') you are asking the network to maximize the Bernoulli likelihood over the training labels.
What is Var(X) for a Bernoulli with p = 0.1?
Discrete and Continuous Probabilities — the MML chapter that formalises the PMF, defines expectation and variance from first principles, and shows how the Bernoulli is the building block for every more complex discrete distribution.
- **PMF**: P(X=1) = p, P(X=0) = 1−p. **Mean** = p, **Var** = p(1−p). Maximum variance 0.25 at p=0.5.
- Binary classification labels are Bernoulli draws. Estimating p given features is the universal binary classification task.
- **Binary cross-entropy = Bernoulli negative log-likelihood.** The loss is not arbitrary; it is derived directly from the probabilistic model.
Logistic regression and binary classifiers output a Bernoulli probability; the loss IS the Bernoulli log-likelihood (binary cross-entropy).
If you remove it: You can't write down the loss function for any binary classification model.