10 · Versioning data with DVC
Git versions your code beautifully — and chokes on a 2 GB dataset. DVC fills the gap: it content-hashes your data, commits a tiny `.dvc` pointer to Git, and pushes the actual bytes to a remote like S3 — so `git checkout` of an old commit brings back the exact data that trained that model.
DVC versions large data the way Git versions code: `dvc add data.csv` CONTENT-HASHES the file and writes a tiny `data.csv.dvc` POINTER (the hash + size) that you commit to GIT; the actual bytes go to a separate REMOTE (S3, GDrive) via `dvc push`. Git stays small and fast, but a `git checkout` of any commit + `dvc pull` brings back the EXACT data that commit referenced.
Without this:
Without data versioning, you can reproduce the code that trained a model but not the DATA — and committing big files directly to Git bloats the repo until clones take forever and the history is unusable.
Git is brilliant at versioning code: small text files, line-by-line diffs, instant history. But ML projects also depend on data and model artifacts — a 2 GB CSV, a 500 MB .joblib, an image folder. Commit those directly to Git and the repository balloons: every version of every big file lives forever in history, clones take ages, and git diff on a binary blob is meaningless. Git was never built for this.
You can reproduce a model only if you can reproduce both the code and the exact data it trained on. So you need data under version control too — just not inside Git. That's exactly what DVC (Data Version Control) provides.
DVC's trick is content-addressing with a pointer:
1. dvc add data.csv — DVC computes a hash (an MD5) of the file's contents and moves the bytes into a local cache keyed by that hash. It writes a tiny text file, data.csv.dvc, containing just the hash and the size — a pointer to the data, not the data itself. It also adds data.csv to .gitignore so Git never tracks the big file directly.
2. You commit the .dvc pointer to Git. That little file (a few hundred bytes) goes into your normal Git history alongside your code. Git stays small and fast; the pointer records which version of the data this commit uses.
3. dvc push sends the actual bytes to a remote — S3, Google Drive, GCS, Azure, an SSH server. The remote is the bulk store; Git holds only pointers.
The payoff is reproducibility. Check out any commit (git checkout v1.2) and Git restores the matching .dvc pointer; dvc pull reads that pointer's hash and fetches the exact data bytes from the remote. Code and data move together, version-locked, without a single big file ever entering Git.
Why does hashing the content matter? Because the hash IS the identity. Identical bytes always produce the identical hash, so DVC never stores the same data twice (deduplication) and instantly knows whether the remote already has a file. Change a single byte and the hash changes completely — a new version. The runnable cell below demonstrates exactly this content-hashing idea with Python's hashlib, which is the kernel of what a .dvc file records.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The DVC workflow: `dvc init`, `dvc add` to hash + track a file (writing a `.dvc` pointer), commit the pointer to Git, `dvc remote add` + `dvc push` the bytes to S3, then `git checkout` + `dvc pull` to restore exact code+data.
When you version a 2 GB dataset with DVC, what goes into Git and what goes into the remote?
- Git versions code but bloats on large files; DVC versions data/model artifacts by content-hashing them and storing only a tiny `.dvc` pointer in Git.
- `dvc add` writes the pointer (hash + size) and gitignores the real file; you commit the pointer to Git and `dvc push` the actual bytes to a remote (S3/GDrive/GCS).
- Content-addressing means identical bytes share one hash (deduplication) and a one-byte change is a new version; `git checkout` + `dvc pull` restores the exact code+data of any commit.
Reproducible ML pipelines version data with DVC (or git-lfs / lakeFS) so a model can always be retraced to the exact dataset that produced it — essential for audits, debugging, and regulatory compliance.
If you remove it: You could reproduce the code that trained a model but not its data — or you'd commit huge files to Git and bloat the repo until it's painful to clone and impossible to diff.