12 · Leak-free evaluation
A recommender is judged per user on what it ranks at the top. Splitting random cells of the matrix leaks the future and ignores time — hold out each user's most recent interaction instead.
Recsys is a ranking problem evaluated per user, not a regression over random cells. A random train/test split lets a model train on a user's future to predict their past — leakage. The honest protocol is leave-one-out (hold out each user's most recent interaction) or a global time cut, training only on what came before.
Without this:
Score your model with a random cell split and you'll report a beautiful number that collapses in production — you measured a model that already saw the answer.
In supervised learning you shuffle rows and split. Recsys breaks both halves of that habit. First, the unit you care about isn't a single rating — it's the ranked list each user actually sees, so you must evaluate per user. Second, interactions happen in time: a user rates today what they'll rate tomorrow only after they've seen it. Both facts forbid the naive split.
Why a random cell split leaks. Pick 20% of the filled cells at random as a test set and train on the other 80%. For a single user, that scatters some of their later interactions into training and some earlier ones into testing. The model effectively trains on the user's future to predict their past — and matrix factorization is especially happy to exploit this: a user/item factor fitted partly on a held-out interaction will of course predict it well. The reported error is optimistic and won't survive contact with real, forward-in-time serving.
The honest protocols:
- Leave-one-out (LOO): for each user, hold out exactly one interaction — their most recent one — as that user's single test 'relevant' item. Train on everything else. Evaluation then asks: does the model rank that held-out item high among all the items the user hadn't touched? This is the standard for ranking metrics (next lesson).
- Global temporal split: pick a cutoff timestamp; everything before is train, everything after is test. The most realistic — it mimics 'train Monday, serve Tuesday' — and the only one that respects causality across all users at once.
Below we take per-user interaction histories with timestamps, build a leave-one-out split (newest interaction per user goes to test), and show explicitly why a random split would have leaked.
Python (in browser)
Leave-one-out by time: each user's most-recent interaction becomes their single test 'relevant' item; everything earlier trains. A random pick leaks the future for most users.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Surprise's LeaveOneOut does the per-user hold-out for you; a true production split sorts on the timestamp column and cuts at a single global moment.
Why is a random cell split (20% of filled ratings held out at random) a leaky way to evaluate a recommender?
- Recsys is evaluated per user on a ranked list — not as a regression over random matrix cells.
- Random cell splits leak the future into training and over-report accuracy.
- Use leave-one-out (hold out each user's most-recent interaction) or a global temporal cut, training only on the past.
Every offline recsys benchmark (MovieLens leave-one-out, RecBole, Spotify's internal eval) uses per-user or temporal hold-outs to avoid leakage.
If you remove it: Without leak-free splitting, offline metrics decouple from live performance and you ship models that look great and serve badly.