9 · Logistic regression: sigmoid + log loss
The simplest classifier: pipe a linear model through a sigmoid to get calibrated probabilities, then train with binary cross-entropy.
Logistic regression = linear regression piped through a sigmoid — a smooth probability output trained with binary cross-entropy.
Without this:
Without it, you can't do classification at all in the linear-model family.
Linear regression predicts a number on the real line — useful for house prices and temperatures, but broken for classification. If you naively use a linear model to predict "spam vs not-spam", two things go wrong:
- Predictions leave [0, 1]. The raw output wᵀx + b can be −∞ or +∞, so you can't interpret it as a probability.
- Squared error is the wrong objective. For binary outcomes (0 or 1), the natural objective is Bernoulli likelihood — not least squares.
Logistic regression fixes both problems.
The key ingredient is the sigmoid function σ(z):
σ(z) = 1 / (1 + e^(−z))
It squashes any real number z to the open interval (0, 1). The model becomes:
P(y=1 | x) = σ(wᵀx + b)
Now the output is a legitimate probability. At z = 0, σ(0) = 0.5 — perfect uncertainty. As z → +∞, σ → 1 (strong positive prediction). As z → −∞, σ → 0 (strong negative prediction).
The log-loss objective (Binary Cross-Entropy)
Given n training samples with targets yᵢ ∈ {0, 1} and predicted probabilities p̂ᵢ, the loss is:
L = −(1/n) · Σ [yᵢ log p̂ᵢ + (1 − yᵢ) log(1 − p̂ᵢ)]
This is derived directly from the negative log-likelihood of a Bernoulli distribution (see Stats track ch7 on MLE). Each sample contributes either log p̂ᵢ (if the true label is 1) or log(1 − p̂ᵢ) (if the true label is 0).
Fitting via gradient descent
Unlike linear regression, there is no closed-form solution for logistic regression because the sigmoid introduces a non-linearity. Instead, the model is trained by gradient descent on the log-loss. The gradient of the loss with respect to the weights has the elegant form:
∂L/∂w = (1/n) · Xᵀ (p̂ − y)
where p̂ is the vector of predicted probabilities and y is the vector of true labels.
The decision boundary
The model predicts class 1 when P(y=1 | x) > 0.5, i.e., when σ(wᵀx + b) > 0.5, i.e., when wᵀx + b > 0. The boundary between the two classes is the hyperplane:
wᵀx + b = 0
In 2D features, this is a line. In 3D, a plane. In higher dimensions, a hyperplane. The decision boundary is always linear in the original feature space — to get curved boundaries you'd need polynomial features or a different model.
Python (in browser)
The sigmoid maps the entire real line into (0, 1). Any wᵀx + b, however extreme, becomes a valid probability. The S-shape means the function is most sensitive near zero — far from the decision boundary it is confident and saturated.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`model.coef_` is the weight vector w; `model.intercept_` is the bias b. `predict_proba` returns the full probability distribution over classes — the second column is P(y=1|x). The two columns always sum to exactly 1.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The gradient from red to green shows model confidence. The white line (prob=0.5) is the linear decision boundary — points to the right are predicted class 1, points to the left class 0. Far from the boundary the sigmoid saturates and confidence is high.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The epsilon clip (`np.clip(p, 1e-12, 1-1e-12)`) prevents log(0) which would give -inf and crash training. Sklearn does this internally. Notice how a confident wrong prediction (y=1 but p_hat=0.1) contributes enormous loss — penalising overconfident mistakes.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
This derivation connects directly to the Stats track (ch7, Bernoulli MLE). If you've worked through that chapter, this is the same maximum-likelihood argument — applied to classification instead of coin flips.
If a logistic regression outputs `predict_proba([[1, 2]])` of `[[0.2, 0.8]]`, the model is:
- P(y=1|x) = σ(wᵀx + b) where σ squashes the linear score to a probability in (0, 1).
- Training minimises binary cross-entropy (log-loss), derived from Bernoulli MLE — no closed form, gradient descent required.
- The decision boundary is the hyperplane wᵀx + b = 0 — always linear in the original feature space.
- On imbalanced data, use `class_weight='balanced'` — otherwise the model learns to predict the majority class.
Default binary classifier in scikit-learn pipelines; the output layer of most binary classification neural networks IS a logistic regression on top of learned features; A/B tests for conversion analysis use it as the standard model.
If you remove it: You'd have to use SVMs or trees from the start — no smooth probabilistic baseline.