Name things for the reader, not the writer.
Name things for the reader, not the writer.
Support Vector Machines
Support Vector Machines (SVMs) solve binary classification by finding the maximum-margin hyperplane that separates two classes. Given labeled data \{(mathbf{x}_n, y_n)\} with y_n in \{-1, +1\}, a hyperplane mathbf{w}^top mathbf{x} + b = 0 classifies by the sign of mathbf{w}^top mathbf{x} + b. Many h
SVM: maximum-margin classifier, solving a convex QP for with margin .
No margin maximization, no support-vector sparsity, no principled reason to prefer one separator over another — generalization bounds fall apart.
Support Vector Machines (SVMs) solve binary classification by finding the maximum-margin hyperplane that separates two classes. Given labeled data with , a hyperplane classifies by the sign of . Many hyperplanes may separate the classes, but SVMs pick a specific one: the one whose distance to the nearest point of either class is largest.
The margin is the distance from the hyperplane to the closest data point. For a normalized constraint , the margin equals . Maximizing the margin is equivalent to minimizing , giving the hard-margin SVM: This is a convex quadratic program — a unique global optimum is guaranteed.
Real data is rarely linearly separable. The soft-margin SVM introduces slack variables to allow violations: The hyperparameter trades off margin size against misclassification: large narrow margin, few violations; small wide margin, more violations allowed.
The dual problem (obtained via Lagrangian duality from Chapter 7) is where SVMs really shine: Two things are magical here: the data appears only through inner products (setting up the kernel trick), and most are zero at the optimum.
Points with are the support vectors — they sit on (or inside) the margin and determine the decision boundary. Non-support vectors could be removed without changing the classifier. This sparsity is the SVM's signature property: the classifier is a weighted sum over a tiny subset of training points, enabling efficient prediction and principled interpretability. Complementary slackness from KKT conditions enforces this structure automatically.
Worked example — hand-computed maximum margin: Three points: with ; with . The widest separator is a vertical line between and — say . Normalize: gives for all three points (all are support vectors). So and the margin width is — the corridor spans from to .
Python (in browser)
Expected: w roughly along (1, 1); margin ≈ 2·√2 / ||w||
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Worked example — effect of shrinking $\mathbf{w}$: If instead you pick (same direction, doubled), then — constraints are satisfied with slack, but is worse than the achieved at . Scaling down to gives functional margin — constraints violated. The SVM optimum is the unique that is just large enough to enforce margin and no larger.
On TF-IDF features, linear SVMs are still state-of-the-art for many text classification tasks — fast to train, easy to interpret, robust to high-dimensional sparse features.
from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import make_pipeline
clf = make_pipeline(TfidfVectorizer(), LinearSVC(C=1.0))Both are 'max-margin' losses in embedding space — push positives close, negatives at least margin away. Same hinge-style geometry as SVM.
Train a SVM to separate the origin from the data in a transformed feature space. Anything that ends up on the 'origin side' of the boundary is flagged.
Put your understanding to the test. Score + streak + speed all count.
3 quick questions. Get 2 right to mark this lesson complete.