11 · Reproducible DVC pipelines
A pile of scripts you run in the right order from memory is not reproducible. A `dvc.yaml` declares the stages — prepare → train → evaluate — with their commands, deps, and outputs, so `dvc repro` re-runs ONLY the stages whose inputs changed, and `dvc dag` draws the whole graph.
A `dvc.yaml` turns ad-hoc scripts into a declared PIPELINE: each STAGE lists its `cmd`, its `deps` (input files/params), and its `outs` (produced files). DVC hashes every dep, so `dvc repro` re-runs ONLY the stages whose inputs actually changed (skipping the rest), `dvc dag` draws the dependency graph, and the whole pipeline becomes a reproducible, cached, version-tracked build.
Without this:
Without a declared pipeline, reproducing a result means remembering which scripts to run in which order — and re-running everything from scratch every time, even the slow stages whose inputs never changed.
Versioning the data (lesson 10) solves what data a model used. But the path from raw data to a trained model is usually several steps — clean the data, engineer features, train, evaluate — and stitching them together by hand is fragile. "Run prepare.py, then train.py, then evaluate.py, but only re-run prepare if you changed the raw data..." That ordering lives in your head and a README, and it breaks the moment someone forgets a step or re-runs the slow training when only the eval changed.
DVC pipelines make that path declarative. You describe your workflow in a dvc.yaml file as a set of stages, and each stage declares three things:
cmd— the command to run (e.g.python src/train.py).deps— the inputs this stage depends on: input data files, the script itself, andparams(named values from aparams.yaml, liketrain.max_depth).outs— the outputs this stage produces: a cleaned dataset, a model file, a metrics JSON. DVC tracks these outputs the same content-addressed way it tracksdvc addfiles.
Once stages are declared, DVC understands the dependency graph: train depends on the output of prepare; evaluate depends on the output of train. That graph is a DAG (directed acyclic graph), and dvc dag draws it for you.
The big win is dvc repro. It walks the DAG and, for each stage, hashes that stage's deps and compares them to the last run. If a stage's inputs are unchanged, DVC skips it and reuses the cached output. Only stages whose deps actually changed — and the stages downstream of them — re-run. Tweak only evaluate.py? DVC reruns just evaluate, reusing the expensive trained model. Change a training param in params.yaml? It reruns train and evaluate but skips prepare. This is exactly how a build tool like make works, but content-hash-driven and data-aware.
So a DVC pipeline gives you three things a folder of scripts never can: a single command to reproduce everything (dvc repro), incremental execution (only what changed re-runs), and a version-tracked, inspectable graph of the whole workflow. DVC isn't browser-runnable, so the dvc.yaml below is read-along — but its structure (cmd / deps / outs per stage) is the entire idea.
A `dvc.yaml` with three stages (prepare → train → evaluate), each declaring `cmd`, `deps`, `outs` (and `params`/`metrics`); the deps form a DAG so `dvc repro` re-runs only changed stages.
Why does a declared `dvc.yaml` pipeline beat running ad-hoc scripts by hand?
- A `dvc.yaml` declares the workflow as stages, each with a `cmd`, its `deps` (inputs/params), and its `outs` (produced files) — turning a pile of scripts into a version-tracked pipeline.
- The deps/outs form a DAG; `dvc repro` re-runs ONLY the stages whose hashed inputs changed (plus downstream stages), reusing cached outputs for the rest.
- `dvc dag` draws the graph and `dvc metrics show` tracks metrics across versions — so the pipeline is reproducible with one command and inspectable, like `make` for ML.
Production ML repos ship a `dvc.yaml` (or an Airflow/Kubeflow DAG) so the data→model path is one reproducible command, re-running only what changed — the foundation CI then automates on every commit.
If you remove it: Reproducing a model would mean remembering an undocumented script order and re-running everything from scratch each time — slow, error-prone, and impossible to hand off cleanly.