8 · Experiment tracking with MLflow
You train a model fifty times with different settings — then someone asks 'which run was best, and how do I reproduce it?'. MLflow turns that from a memory game into a query: every run records its params, metrics, and artifacts so the answer is a click, not a guess.
MLflow tracking wraps each training attempt in a RUN that records its parameters (the knobs you set), its metrics (how well it did), and its artifacts (the model file, plots, the trained pipeline) — so 'which run was best?' and 'how do I reproduce it?' become a query against a structured log instead of a search through your memory and scattered notebooks.
Without this:
Without tracking, every experiment lives in volatile state — a printed number, a notebook cell you overwrote, a filename like `model_v3_final_FINAL.joblib`. You can't compare runs, can't say which hyperparameters won, and can't reproduce last week's best result.
Training a model is rarely one shot. You try a learning rate, then another; you add a feature, swap a model, tune max_depth. After dozens of attempts the inevitable question arrives: which run was best — and how do I reproduce it? If your only record is printed accuracy numbers scrolling past in a terminal and a folder of model_final_v2.joblib files, the honest answer is "I'm not sure." MLflow exists to make that answer a query instead of a guess.
The core abstraction is the run. A run is one execution of your training code, and around it MLflow records four kinds of things:
1. Parameters — the inputs you chose: the learning rate, n_estimators, the train/test split seed, which feature set. These are the knobs. You log them with mlflow.log_param("max_depth", 8). Params are how you later ask "what settings produced this result?"
2. Metrics — the outputs that measure quality: accuracy, F1, AUC, validation loss. Logged with mlflow.log_metric("val_auc", 0.91). Metrics can be logged repeatedly with a step (e.g. loss per epoch), so MLflow stores a curve, not just a final number.
3. Artifacts — the files a run produces: the serialized model (model.joblib), a confusion-matrix plot, a feature-importance chart, even the input dataset. Logged with mlflow.log_artifact("plots/confusion.png"). The trained model itself is the most important artifact — mlflow.sklearn.log_model(...) saves it in a loadable format.
4. The run's identity — MLflow automatically stamps each run with a unique ID, a start time, the source code version (git commit), and groups runs into named experiments.
You wrap your training in with mlflow.start_run(): — everything logged inside belongs to that run. Then the tracking UI (mlflow ui, a local web app) shows every run as a row in a sortable, filterable table: sort by val_auc descending and the best run is the top row, with its exact params right there. Autolog (mlflow.sklearn.autolog()) goes further — it automatically captures params, metrics, and the model for supported libraries with a single line, no manual log_* calls.
MLflow needs a tracking server / local store and isn't a browser thing, so the code below is read-along — but the shape is what matters: wrap training in a run, log params + metrics + the model, query later.
A training script wrapped in `mlflow.start_run()`: log params, log metrics, log the model artifact — plus `mlflow.sklearn.autolog()` to capture all of it in one line.
What does a single MLflow run record?
- An MLflow run wraps one training attempt and records its parameters (inputs), metrics (quality), and artifacts (model file, plots) — with an auto-stamped ID, time, and code version.
- Wrap training in `with mlflow.start_run():` and log params/metrics/the model; the tracking UI then makes 'which run was best?' a sortable, filterable query.
- `mlflow.sklearn.autolog()` captures params, metrics, and the model automatically for supported libraries in a single line — no manual `log_*` calls needed.
Every serious ML team runs an experiment tracker (MLflow, Weights & Biases, Neptune). It's how you compare hundreds of runs, defend 'this is our best model' with evidence, and reproduce a result months later.
If you remove it: Your experiments would live in volatile state — printed numbers and overwritten files — so you couldn't reliably compare runs, justify a model choice, or reproduce a past result.