7 · Cross-validation
One train/test split gives you one score; k-fold gives you a confidence interval — and uses your data k times more efficiently.
Train/test split gives you one number; k-fold cross-validation gives you a confidence interval — and uses your data k times more efficiently.
Without this:
Without cross-validation, you'd over-trust a lucky 80/20 split and ship models whose 'good' test score was noise.
A single train/test split is the simplest evaluation protocol — but it has a hidden problem: the score you get depends heavily on which rows happened to land in the test set. If you're unlucky, your test set is unusually easy or unusually hard. The variance of a single split is high.
The train / val / test distinction
Three-way split protocol:
- Training set — model sees this data during fitting.
- Validation set — used to tune hyperparameters and compare models. The model never trains on this.
- Test set — touched once, at the very end, to report the final honest score. Peeking at it during development invalidates the evaluation.
k-fold cross-validation
Instead of one split, k-fold divides the data into k equal chunks (folds). For each fold i:
- Train on all folds except i.
- Validate on fold i.
- Record the metric.
Rotate until every fold has been the validation fold exactly once. You end up with k scores — report the mean and standard deviation. Common choices: k = 5 or k = 10.
Advantages over a single split:
- Every row is used for both training and validation (across different rounds).
- The k scores let you report a mean ± std — a rough confidence interval.
- Much more stable estimate of generalisation performance.
Stratified k-fold — for classification: preserves class proportions in each fold. Critical for imbalanced datasets where a naive split might put all minority-class examples in one fold.
Leave-one-out CV (LOOCV) — special case where k = n. Maximally uses data; gives an unbiased estimate; computationally expensive (n model fits).
TimeSeriesSplit — for sequential/temporal data: folds must respect chronological order. Fold i trains on rows 0..m×i and validates on rows m×i..m×(i+1). Shuffling would leak future information into training — a form of data leakage.
Python (in browser)
`cross_val_score` handles the split, fit, and score loop automatically. The std across folds tells you how sensitive the model's performance is to which rows end up in validation.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The manual loop and `cross_val_score` produce identical results when given the same `KFold` splitter. The manual loop is useful when you need access to the fitted model inside each fold (e.g., to inspect feature importances).
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Without stratification some folds may receive 0 or 1 minority examples — making the fold's metric meaningless. StratifiedKFold ensures each fold reflects the overall class balance.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
TimeSeriesSplit never shuffles. Fold boundaries march forward in time — training always ends before validation begins. This is the only valid CV strategy when rows are ordered by time.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Nested CV is the statistically honest approach to combined model selection and evaluation. In practice, even one level of CV is a large improvement over a single train/test split.
On a time-series forecasting problem, can you use KFold with shuffle=True?
- k-fold CV gives k scores; report mean ± std as a rough 95% confidence interval on test performance.
- Always set shuffle=True (with a fixed random_state) for KFold on non-time-series data.
- Use StratifiedKFold for classification — especially with imbalanced classes.
- Use TimeSeriesSplit (never KFold) for temporal data — future rows must never appear in the training fold.
Every Kaggle solution reports CV scores. Sklearn's GridSearchCV and RandomizedSearchCV use CV under the hood. Hyperparameter tuning frameworks (Optuna, Ray Tune) all rely on a CV-based objective.
If you remove it: You'd report performance metrics that are 'lucky' rather than reliable — a recipe for production surprises.