scikit-learn SVC with RBF
sklearn
The default tabular non-linear baseline. Works well on small/medium datasets where deep learning would overfit.
python
from sklearn.svm import SVC
clf = SVC(kernel='rbf', C=1.0, gamma='scale').fit(X, y)Ch. 12 · Support Vector Machines
SVM: Kernels
Support Vector Machines
Ch. 12
yi(w⊤xi+b)≥1
the max-margin constraint
Linear SVMs can only draw straight decision boundaries — yet most interesting data is non-linearly separable. The kernel trick turns linear SVMs into universal classifiers without ever computing features explicitly. The insight comes from the dual: it depends on mathbf{x}_n only via inner products m
⏱ 2min read◇ 2vizs.✓ 5exercises
Before you start
Foundations you'll lean on here
A quick refresher on what this lesson assumes. Tap any card for a plain-language explanation and a tiny worked example.
Concepts you'll lean on
Tap any to see its definition and a worked example.
★Big idea
The kernel trick replaces with — implicit feature lifting without explicit computation.
Linear SVMs can only draw straight decision boundaries — yet most interesting data is non-linearly separable. The kernel trick turns linear SVMs into universal classifiers without ever computing features explicitly. The insight comes from the dual: it depends on only via inner products . Replace every such inner product with a kernel function , and you are implicitly operating in a different feature space such that — without ever computing .
The kernelized dual is and the decision function is a kernel-weighted sum over support vectors. A valid kernel must be positive semi-definite (Mercer's condition): for any , the Gram matrix must be PSD. Only such kernels correspond to genuine inner products in some feature space.
Three kernels power 90% of applications. Polynomial implicitly maps into degree- polynomial features — cheap and interpretable. Radial Basis Function (RBF) maps into an infinite-dimensional feature space and is the default choice for most problems. Sigmoid (not always PSD — careful) gives a shallow neural network flavor.
Lifting to a higher dimension
Data that is not linearly separable in 2D becomes separable in a higher-dimensional feature space. The kernel computes the inner product there without forming the features.
↻
Kernel choice and hyperparameters are the main modeling decisions. The RBF bandwidth controls smoothness: small gives a highly flexible, wiggly boundary (risk of overfitting); large gives a nearly linear boundary (risk of underfitting). Combined with the penalty , RBF SVMs have two hyperparameters — — which are usually selected by cross-validation over a log-spaced grid. This simplicity and strong regularization is why RBF SVMs were the state of the art on many tasks for decades.
Beyond SVMs, kernels power kernel ridge regression, Gaussian processes, kernel PCA, and kernel $k$-means. They illustrate a profound principle: any algorithm that uses data only via inner products can be kernelized. The cost is memory for the Gram matrix, which limits classical kernels to tens of thousands of points. Modern scalable variants — Nystrom approximations, random features (Rahimi-Recht), and neural tangent kernels — preserve the kernel mindset while scaling to millions of samples.
Worked example — polynomial kernel computation: , kernel . Inner product: . Kernel value: . The implicit feature map is — 6-dimensional. Verify: , , and . ✓
1/2
RBF kernel vs linear on the moons dataset
Python (in browser)
Expected: RBF significantly outperforms linear on this non-linearly-separable dataset
main.pypython
Loading editor…
Ready — click Run to execute in your browser.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Non-linear decision boundary via RBF
An RBF-kernel SVM draws a curved boundary in input space — which is a linear boundary in the implicit feature space.
Worked example — RBF kernel computation: Same , kernel (so ). . So . If we halve to , the kernel becomes — a smaller bandwidth rapidly zeros out similarity between distant points, producing a more wiggly, local decision boundary.
1/3
In the wild
Where this shows up in production
Each card is a real system, model, or technique that rests on this lesson.
Graph kernels for molecules / proteins
GraphKernelChemoinformatics
Weisfeiler-Lehman and graph-edit kernels let you SVM-classify molecular graphs without learning a representation — still competitive on small chemistry datasets.
Kernel methods inside transformers (Performer)
PerformerTransformers
FAVOR+ replaces softmax attention with a random-feature kernel approximation, making attention linear in sequence length.
Exercises
Concept Challenge
Put your understanding to the test. Score + streak + speed all count.
SVM: Kernels
5 rounds1000 max pts
Active recall
Recap flashcards
1 / 3
Concept
The kernel trick replaces with
Tap the card to reveal
Confirm you've got it
3 quick questions. Get 2 right to mark this lesson complete.