16 · Naive Bayes: Gaussian, Multinomial & Bernoulli
Apply Bayes' theorem with one simplifying assumption — feature independence given the class — and get a blazing-fast classifier that dominates text problems.
Apply Bayes' theorem assuming features are conditionally independent given the class — fast, surprisingly competitive, and the spam-filter workhorse.
Without this:
Without it you'd skip a classifier that's blisteringly fast and especially strong for text and document classification.
Most classifiers learned so far are discriminative — they learn a boundary P(y|x) directly from data. Naive Bayes flips the script and is generative: it models how data is generated per class, then uses Bayes' theorem to infer the class from the data.
Bayes' theorem applied to classification
For a new point x and a class label y:
P(y | x) = P(x | y) · P(y) / P(x)
Since P(x) is the same for all classes, the decision rule simplifies to picking the class that maximises the numerator:
ŷ = argmax_y P(x | y) · P(y)
- P(y) is the prior — the class frequency in training data.
- P(x | y) is the likelihood — how probable are these feature values for class y?
- P(y | x) is the posterior — what we actually want.
The "naive" assumption
Computing P(x | y) for a full d-dimensional vector x requires estimating a d-dimensional joint distribution — exponentially expensive. Naive Bayes sidesteps this by assuming features are conditionally independent given the class:
P(x | y) = P(x₁ | y) · P(x₂ | y) · … · P(xd | y) = ∏ P(xᵢ | y)
This is almost never literally true, but the simplification makes training O(n · d) — trivially fast — and predictions are often surprisingly competitive.
Three sklearn variants
| Variant | Assumption about P(xᵢ | y) | Typical use | |---|---|---| | GaussianNB | Each feature follows a 1-D Gaussian per class | Continuous features (e.g. Iris, medical) | | MultinomialNB | Feature counts follow a Multinomial distribution | Word counts, TF vectors | | BernoulliNB | Binary features follow a Bernoulli distribution | Binary bag-of-words, spam detection |
Bias–variance angle
The independence assumption introduces high bias (the model is wrong about correlations), but because the model has very few parameters (2 per feature per class for GaussianNB), variance is low and predictions are stable. On small datasets this bias-variance trade-off often makes Naive Bayes competitive with far more complex models.
Lesson mml-34 covers the Sum Rule, Product Rule, and Bayes' theorem from first principles — the exact identities that underpin Naive Bayes. Review it if the P(y|x) = P(x|y)·P(y)/P(x) derivation feels shaky.
Python (in browser)
`model.theta_` stores the per-class feature means μ_{y,j}; `model.var_` stores the per-class variances σ²_{y,j}. These two arrays completely define the GaussianNB likelihood: P(xᵢ | y) = N(xᵢ; μ_{y,i}, σ²_{y,i}). Training is simply computing column means and variances per class — O(n·d).
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`MultinomialNB` models each feature (word) as a count drawn from a multinomial distribution parameterized per class. `alpha` is the Laplace smoothing constant — add `alpha` pseudo-counts to every word before computing log probabilities. `alpha=1.0` is the most common default; try `alpha=0.1` for sparser smoothing on large vocabularies.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
GaussianNB's decision boundary is a conic section (parabola / ellipse) because it models class-conditional Gaussians. When both classes have the same covariance the boundary degenerates to a straight line — identical to LDA and close to logistic regression. The key trade-off: NB is nearly instantaneous to train and nearly instantaneous to update with new data (online learning), while LR needs an iterative solver.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The independence assumption causes NB to double-count correlated evidence, but the class ranking is often still correct. This is why NB accuracy can be good even when its probability outputs are not well-calibrated.
If two features are perfectly correlated, Naive Bayes will:
- Naive Bayes applies Bayes' theorem with the conditional independence assumption: P(x|y) = ∏ P(xᵢ|y). Training is O(n·d) — as fast as it gets.
- GaussianNB models continuous features as per-class Gaussians; MultinomialNB uses count distributions; BernoulliNB handles binary features.
- Correlated features cause NB to double-count evidence — probabilities become overconfident but rankings are often still correct.
- NB probabilities are NOT calibrated. Use CalibratedClassifierCV if you need trustworthy probability scores, not just class predictions.
- MultinomialNB with Laplace smoothing (alpha=1.0) is a strong, fast baseline for text classification — always try it before heavier models.
Email spam filters; document categorisation; sklearn's default baseline for text problems; the conceptual ancestor of modern generative classifiers.
If you remove it: You'd miss the fastest-training classifier — useful for prototype baselines and online learning.