13 · SVM classification: hard + soft margin
Find the widest possible gap between classes, penalise violations with slack variables, and discover why only the support vectors matter.
SVM finds the WIDEST possible margin between classes — and the points on the boundary (support vectors) are the only ones that matter.
Without this:
Without SVM you lose the most elegant geometric classifier and the gateway to kernel methods.
Logistic regression and SVM are often called "geometric cousins": both find a linear boundary that separates two classes, and both can be regularised. But their philosophies are very different.
Logistic regression maximises the likelihood of the training labels — it is indifferent to where the boundary sits as long as the probabilities are correct.
SVM is obsessed with geometry. It finds the hyperplane that maximises the margin — the perpendicular distance between the boundary and the nearest training points.
The separating hyperplane
A linear classifier defines a hyperplane:
wᵀx + b = 0
Points on one side satisfy wᵀx + b > 0 (class +1); points on the other side satisfy wᵀx + b < 0 (class −1).
The margin
After normalising w so that the nearest training points satisfy |wᵀx + b| = 1, the two parallel "margin boundaries" are:
wᵀx + b = +1 (nearest positive points) wᵀx + b = −1 (nearest negative points)
The perpendicular distance between these two planes is 2/||w||. Maximising the margin is equivalent to minimising ½||w||² — a convex quadratic program with a unique solution.
Hard margin — rarely realistic
If the data is perfectly linearly separable, the hard-margin SVM finds the boundary that keeps ALL points strictly outside the margin zone. In practice, real data almost always has overlap or outliers, and a hard-margin SVM either fails to converge or overfits.
Soft margin — the practical SVM
Introduce slack variables ξᵢ ≥ 0, one per training point. A point is allowed to violate the margin by ξᵢ units. The objective becomes:
Minimise ½||w||² + C·Σξᵢ
subject to yᵢ(wᵀxᵢ + b) ≥ 1 − ξᵢ and ξᵢ ≥ 0.
The hyperparameter C controls the trade-off:
- Large C → small margin, low tolerance for misclassification (low bias, high variance)
- Small C → large margin, more tolerance (high bias, lower variance)
The dual formulation and support vectors
Via Lagrangian duality, the solution depends only on training points with non-zero dual variable αᵢ. These are exactly the points ON the margin (ξᵢ = 0) or VIOLATING it (ξᵢ > 0). They are called support vectors because they "support" (define) the decision boundary. Remove any other training point and the solution does not change.
Python (in browser)
The solid yellow line is the decision boundary (wᵀx + b = 0). The dashed grey lines are the margin boundaries (wᵀx + b = ±1). The circled points are the support vectors — the only training samples that determine the boundary. Every other point could be moved (as long as it stays outside the margin) without changing the solution.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Left: C = 0.1 — the model tolerates many violations to keep a wide, robust margin. Right: C = 100 — the model fights hard to classify every point correctly, shrinking the margin. A tight margin on overlapping data is a recipe for poor generalisation.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`model.support_vectors_` contains the actual feature values of the support vectors. `model.support_` contains their original indices in X. `model.n_support_` is the count per class. Only these points determine w and b — the rest of the training data is irrelevant once the model is fit.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The SVM objective is a quadratic program — guaranteed globally convex. The C·Σξᵢ term is the hinge loss in disguise. Notice the dual form only uses dot products xᵢᵀxⱼ — replacing those with a kernel function K(xᵢ, xⱼ) gives non-linear SVMs without ever computing the high-dimensional feature map.
The 'support vectors' in an SVM are:
- The margin = 2/||w||; maximising the margin is equivalent to minimising ½||w||² — a unique convex QP.
- Soft-margin SVC introduces slack variables ξᵢ; C controls the penalty — high C = tight margin, low C = wide margin.
- Only support vectors (αᵢ > 0) determine the decision boundary — all other training points are irrelevant after fitting.
- Don't use vanilla SVC on more than ~50k samples — it scales O(n²) to O(n³). Use LinearSVC for large linear problems.
Text classification with high-dimensional sparse features (SVM was the gold standard before deep learning). Anomaly detection via one-class SVM. Sklearn examples and tutorials still feature SVM heavily as the second-most-common baseline after logistic regression.
If you remove it: You'd miss the conceptual stepping stone from linear models to kernel methods — and the most data-efficient classifier for small/medium tabular problems.