15 · Support Vector Regression (SVR)
The SVM idea applied to regression: find the flattest function inside an ε-tube, ignore small errors, and penalise only what sticks out.
SVR finds the FLATTEST function inside an ε-tube around the data — robust to outliers and naturally regularised.
Without this:
Without SVR you lose a kernelised regressor that handles non-linearity AND outliers gracefully.
SVM classification asks: "find the widest margin that separates two classes." SVR asks the parallel question: "find the flattest function that fits the data within a tolerance band."
The ε-insensitive loss
Instead of penalising every prediction error (as in squared loss), SVR defines a "tube" of width ε around the fitted function. Errors smaller than ε contribute NOTHING to the loss. Only points that fall outside the tube are penalised — linearly, not quadratically.
This is the ε-insensitive loss:
L_ε(y, ŷ) = max(0, |y − ŷ| − ε)
Compare to:
- Squared loss: (y − ŷ)² — every error hurts, large errors hurt a lot
- ε-insensitive: zero below ε, linear above ε — like Huber loss but with a dead zone
The SVR objective
Minimise ½||w||² + C·Σ(ξᵢ + ξᵢ*)
subject to: yᵢ − (wᵀxᵢ + b) ≤ ε + ξᵢ (upper slack) (wᵀxᵢ + b) − yᵢ ≤ ε + ξᵢ* (lower slack) ξᵢ, ξᵢ* ≥ 0
The ½||w||² term drives the function to be flat (low complexity). Points inside the tube have ξᵢ = ξᵢ* = 0. Points outside become support vectors.
Key hyperparameters
- C — penalty for being outside the tube (same trade-off as SVC: high C = low tolerance, tight fit; low C = more slack)
- ε — tube width. Large ε = fewer support vectors, coarser fit. Small ε = more support vectors, tighter fit.
- kernel/γ — same as SVC: choose the non-linearity level
When SVR shines
Use SVR when: dataset is small-to-medium (< 50k points), the relationship is non-linear, and you expect outliers or label noise. The ε-insensitive loss gives natural robustness without manual outlier removal.
Python (in browser)
The linear kernel can only fit a straight line — it misses the sinusoidal pattern completely. Polynomial (d=3) captures some curvature but can diverge at the edges. RBF tracks the true sin(x) closely while smoothing over the noise — the ε-insensitive loss prevents overfitting to individual noisy points.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The green shaded band is the ε-tube. Points inside the band (grey) contribute zero to the loss — the model ignores them. Red points fall outside the tube and become support vectors — they determine the shape of the fitted curve. A wider ε means fewer support vectors and a simpler (flatter) model.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
With outliers injected, `LinearRegression` and `Ridge` are pulled toward the outliers (squared loss amplifies them). SVR with ε-insensitive loss gives them only linear weight — the influence is bounded. SVR-rbf wins the cross-validation MSE comparison by a clear margin.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
SVR occupies a useful niche: non-linear regression on small datasets where you can't afford gradient boosting's appetite for data. The ε-tube is the key differentiator — it builds in outlier robustness that Ridge and plain OLS don't have.
In SVR, points INSIDE the ε-tube contribute:
- SVR uses the ε-insensitive loss: zero penalty inside the tube, linear penalty outside — robust to outliers by design.
- Points inside the ε-tube contribute nothing to the loss and are not support vectors — sparsity is built in.
- C controls fitting tightness; ε controls tube width; gamma (for RBF) controls the reach of each support vector.
- Always StandardScaler before SVR; use SVR for small/medium non-linear regression with outliers — switch to GBM or neural nets for large datasets.
Financial returns prediction (small data + outliers); structural engineering response surfaces; biological dose-response curves.
If you remove it: You'd default to linear regression or non-robust polynomial fits and get punished by outliers.