12 · GridSearchCV + RandomizedSearchCV
Stop fiddling with hyperparameters by hand — automate the search with CV-based grid and random search.
Picking hyperparameters by hand is amateur — automate the search with CV-based grid/random search.
Without this:
Without automated tuning, your model is probably 5-10% below its potential — and you'll burn hours of manual fiddling to close the gap.
Every machine learning model has two kinds of quantities:
Parameters — values learned during training by minimising the loss function. In logistic regression, the weights w and bias b are parameters. The algorithm finds them; you don't set them.
Hyperparameters — values set before training that control the learning process. Examples:
- Regularisation strength
Cin logistic regression (C = 1/λ) - Penalty type (
l1vsl2) - Learning rate, number of trees, max depth in other models
Choosing hyperparameters is a meta-optimisation problem: you want the hyperparameter values that maximise generalisation performance (CV score), not training score.
GridSearchCV — exhaustive search
Define a grid of candidate values for each hyperparameter. GridSearchCV tries every combination with k-fold cross-validation and selects the one with the best mean CV score.
Example: if you search C ∈ {0.01, 0.1, 1, 10, 100} and penalty ∈ {l1, l2}, that is 5 × 2 = 10 combinations × 5 folds = 50 model fits.
Strength: guaranteed to find the best configuration in the grid. Weakness: exponential in the number of hyperparameters. 10 parameters × 5 values each = 9.7 million combinations — computationally impossible.
RandomizedSearchCV — random sampling
Instead of an exhaustive grid, sample n_iter combinations randomly from the hyperparameter distributions. Each combination is evaluated with k-fold CV.
Key insight (Bergstra & Bengio 2012): when most hyperparameters don't matter, random search is much more efficient than grid search. In a 2D grid where one parameter matters and one doesn't, grid search wastes most evaluations — random search keeps trying new values of the important parameter.
Modern alternatives: Bayesian optimisation
Tools like Optuna (TPE algorithm) and scikit-optimize maintain a probabilistic model (Gaussian process or Tree Parzen Estimator) of the validation surface. Each new sample is chosen to maximise expected improvement — sampling more densely where the surface looks promising. Much faster than random search for high-dimensional hyperparameter spaces (>5 parameters).
The right evaluation protocol: Nested CV
Hyperparameter search introduces its own overfitting: if you use the same CV scores to select hyperparameters AND to report final performance, the final score is optimistic (it was used to pick the model). The correct approach is nested CV: an outer loop estimates true generalisation, an inner loop selects hyperparameters.
Python (in browser)
Using `Pipeline` with `GridSearchCV` is the correct pattern — the scaler is re-fit on each training fold, preventing data leakage from the test fold's statistics. Note the `clf__C` naming convention: double underscores separate the Pipeline step name from the parameter name.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`scipy.stats.loguniform(1e-3, 1e3)` samples C uniformly on a log scale — so it tries values like 0.002, 0.15, 3.7, 47, etc. This covers the full range without the artificial discretisation of a grid. For the same number of fits, Randomized often matches or beats Grid.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Nested CV is the statistically rigorous approach. In practice even single-loop CV-based tuning is a massive improvement over no tuning at all — just be aware that the reported score is slightly optimistic.
If you have 4 hyperparameters with 10 values each and 5-fold CV, how many model fits does GridSearchCV perform?
- Parameters are learned by the optimizer; hyperparameters are set before training and tuned with CV.
- GridSearchCV is exhaustive (guarantees finding the grid best) but exponential in the number of hyperparameters.
- RandomizedSearchCV samples continuous distributions — often matches or beats Grid for the same compute budget, and scales to more hyperparameters.
- Never tune hyperparameters on the test set — always keep a clean held-out set for final evaluation.
Every winning Kaggle solution involves hyperparameter tuning — usually with Optuna or LightGBM/XGBoost's built-in tuners. Production ML pipelines have a 'tuning stage' that runs nightly or weekly.
If you remove it: You'd ship models with arbitrary hyperparameter choices — and a 5-10% lift typically waits behind a 30-minute tune.