11 · Multiclass classification: OVR + softmax
Go beyond binary: One-vs-Rest trains K binary classifiers; softmax trains one model that outputs K calibrated probabilities summing to 1.
Two strategies to go beyond binary: train one binary classifier per class (One-vs-Rest), or output one normalized probability per class (softmax).
Without this:
Without multiclass support, you couldn't do digit classification, language identification, or any K-class problem.
Binary logistic regression handles exactly two classes. Real problems often have K > 2 classes — handwritten digits (0-9), species identification (hundreds of classes), language identification (dozens of languages). Two strategies extend logistic regression to K classes:
OVR — One-vs-Rest (also called One-vs-All)
Train K separate binary classifiers. Classifier k asks "is this sample class k, or something else?" At prediction time, run all K classifiers and pick the class with the highest predicted probability.
Pros: simple; each sub-model can be trained independently (parallelisable). Cons: each sub-classifier sees an imbalanced dataset (1 positive class vs K-1 negative); K models to store and maintain.
OVO — One-vs-One
Train K(K-1)/2 binary classifiers — one for every pair of classes. At test time, each classifier votes and the class with the most votes wins.
Pros: each model is trained on a balanced 2-class problem. Cons: K(K-1)/2 models — expensive for large K (e.g., K=100 → 4950 models). Less common in practice.
Softmax (Multinomial Logistic Regression)
A single model that directly outputs K calibrated probabilities summing to 1. The model learns K weight vectors (one per class) and applies the softmax transformation:
softmax(z)ₖ = e^zₖ / Σⱼ e^zⱼ
where zₖ = wₖᵀx + bₖ is the raw score (logit) for class k. The output is a proper probability distribution: all values in (0,1), all sum to 1.
The loss is categorical cross-entropy — the natural generalisation of BCE to K classes:
L = −(1/n) · Σᵢ Σₖ yᵢₖ · log(p̂ᵢₖ)
where yᵢₖ = 1 if sample i is class k (one-hot encoding), else 0.
In sklearn, LogisticRegression(multi_class="multinomial", solver="lbfgs") uses softmax and categorical cross-entropy. Since sklearn 0.22+, multinomial is the default when the solver supports it.
Python (in browser)
Every row of `predict_proba` sums to exactly 1.0 — softmax guarantees a valid probability distribution across classes. The predicted class is always the column with the highest probability.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
On well-separated classes like Iris, OVR and multinomial give similar accuracy. On harder problems with overlapping classes, softmax often wins because it optimises a single joint loss over all classes simultaneously instead of K independent losses.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The `z - z.max()` trick (subtracting the maximum logit before exponentiating) keeps all values in exp() non-positive, preventing float overflow. The result is mathematically identical because the max cancels in the fraction — but numerically much safer.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The rule is simple: mutually exclusive classes → softmax; overlapping/multi-label classes → OVR (or a dedicated multi-label model).
For a 4-class problem, softmax outputs (0.1, 0.4, 0.3, 0.2). What class does the model predict?
- OVR trains K binary classifiers and picks the argmax at prediction time — simple and parallelisable.
- Softmax produces a valid probability distribution over K classes from K weight vectors; each row sums to 1.
- Always subtract z.max() before computing softmax to avoid exp() overflow.
- Softmax + categorical cross-entropy is the universal output layer for multi-class neural networks.
Image classification (ImageNet 1000 classes uses softmax); language model token prediction (vocabulary of 50k+ tokens via softmax); intent classification in NLU.
If you remove it: You'd be stuck with binary classifiers and inelegant patches when you need K-class output.