14 · SVM kernels: linear, polynomial, RBF, sigmoid
The kernel trick maps data into high-dimensional space implicitly — giving SVMs non-linear boundaries without paying the computational cost.
The kernel trick replaces the dot product with a function K(xᵢ,xⱼ) that implicitly maps data into a higher-dimensional space where it becomes linearly separable.
Without this:
Without the kernel trick SVMs would be stuck at linear boundaries — and so would Gaussian processes, kernel PCA, and kernel ridge regression.
The previous lesson showed that the SVM dual objective only uses training points through dot products xᵢᵀxⱼ. The kernel trick exploits this: instead of computing an explicit high-dimensional feature map ϕ(x), you replace every dot product with a kernel function:
K(xᵢ, xⱼ) = ϕ(xᵢ) · ϕ(xⱼ)
The magic is that for carefully chosen K, you can compute this inner product in the ORIGINAL space cheaply — even though ϕ maps to an infinite-dimensional space. You never materialise ϕ(x) explicitly.
Four canonical kernels
Linear: K(x, y) = xᵀy The degenerate case — no feature map, same as the previous lesson. Use when n_features >> n_samples (e.g., text classification with TF-IDF vectors).
Polynomial: K(x, y) = (γ xᵀy + r)^d Implicitly computes all monomial features of degree up to d. Use when you have reason to believe the interactions between features matter (e.g., image pixels, combinatorial biology data).
RBF (Radial Basis Function / Gaussian): K(x, y) = exp(−γ ||x − y||²) The universal default. Can approximate ANY continuous function given enough support vectors. Small γ → broad, smooth influence (features more global); large γ → narrow influence (each support vector creates a localised "island").
Sigmoid: K(x, y) = tanh(γ xᵀy + r) Historically motivated by neural networks (one hidden layer with tanh activations). In practice, it often fails Mercer's condition (not always positive semi-definite) and is rarely used in modern ML.
Choosing γ
For the RBF kernel, γ controls the "reach" of each support vector:
- Small γ → smooth decision boundary, risk of underfitting
- Large γ → tight decision boundary around each support vector, risk of overfitting
- Rule of thumb: start with
gamma='scale'(sklearn's default: 1/(n_features · Var(X))) and tune from there.
Python (in browser)
The linear kernel fails entirely on the non-linear moon-shaped data. Polynomial (d=3) starts to bend the boundary. RBF fits the curved shape naturally. Sigmoid performs poorly here — its non-Mercer behaviour produces erratic boundaries on low-dimensional data.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
From left to right, gamma increases from 0.01 to 10. Small gamma produces a smooth, generalising boundary. Large gamma creates isolated 'islands' around individual support vectors — a classic sign of overfitting. Note how the support vector count DECREASES as gamma increases (each SV covers a smaller neighbourhood so fewer are needed to memorise the training set).
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
This is the standard SVM workflow in practice: wrap in a Pipeline (scaler + SVM), then GridSearchCV over C and gamma. Always include `StandardScaler` — RBF distances are meaningless on unscaled features. The Pipeline ensures the scaler is fit only on training folds (no data leakage).
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Kernel selection is a domain decision, not a tuning decision. Pick the kernel based on your data structure, then tune C and gamma with cross-validation. The kernel trick generalises beyond SVMs — kernel PCA, kernel ridge regression, and Gaussian processes all use the same K(xᵢ, xⱼ) substitution.
Large gamma in an RBF SVM leads to:
- The kernel trick replaces dot products with K(xᵢ, xⱼ) = ϕ(xᵢ)·ϕ(xⱼ) — you get high-dimensional feature maps for free.
- RBF is the default non-linear kernel; linear is preferred for high-dimensional sparse data; polynomial when domain knowledge suggests it; almost never sigmoid.
- Small gamma = smooth, global boundary; large gamma = localised, overfitting-prone boundary. Always tune gamma with cross-validation.
- ALWAYS StandardScaler before RBF SVM — distance-based kernels are dominated by features with large numeric ranges.
Pre-deep-learning era: SVM with RBF was the workhorse for tabular ML. Today: kernel ridge regression remains a strong baseline; Gaussian processes use the same kernels for Bayesian regression; transformer attention is conceptually 'learned soft kernels'.
If you remove it: Linear SVM would only handle linearly separable problems — most real classification is non-linear.