17 · Monitoring & drift detection
A deployed model doesn't fail loudly — it silently rots as the world shifts away from its training data. Log every prediction, then watch for drift: DATA drift (the inputs change) and CONCEPT drift (the X→y relationship changes). A metric like PSI quantifies the shift and trips a retraining alert.
A deployed model degrades silently: it keeps returning predictions even as the live data drifts away from what it was trained on. Monitoring catches this. You LOG every prediction (inputs + outputs), then watch for two kinds of drift — **DATA drift** (the input distribution P(X) shifts: a feature's mean or spread moves) and **CONCEPT drift** (the relationship P(y|X) shifts: the same inputs now map to different outcomes). A drift metric like the **PSI (Population Stability Index)** quantifies how far the current input distribution has moved from the training reference; cross an alert threshold (a common rule: PSI > 0.2) and you trip a **retraining trigger**. Monitoring is the smoke detector that turns silent decay into an actionable alarm.
Without this:
Without monitoring, a model's accuracy erodes invisibly — there's no error, no crash, just gradually worse predictions on a world that has moved on — and you only find out when a business metric tanks or a user complains, long after the damage is done.
Every prior lesson got the model running. This one keeps it honest. A deployed model has a failure mode unlike normal software: it doesn't crash, it doesn't throw, it just silently gets worse. The code is fine; the world moved. Your fraud model was trained on last year's fraud patterns, but fraudsters changed tactics. Your demand forecaster learned pre-pandemic behavior, then everyone's habits shifted. The model keeps confidently returning predictions — they're just increasingly wrong. This silent rot is why monitoring is the last, non-optional stage of MLOps.
Step one: log every prediction. At serving time, record each request's inputs and the prediction (and, when ground truth eventually arrives, the actual outcome). This log is the raw material for everything else — without it you're blind. Operational metrics matter too (latency, error rate, throughput), but the ML-specific signal is in the data and predictions themselves.
The two kinds of drift. The world can shift in two distinct ways:
- Data drift — the input distribution P(X) changes. The kind of data flowing in is no longer what you trained on: a feature's mean creeps up, a category that was rare becomes common, a sensor recalibrates. The X→y relationship might still hold, but the model is now extrapolating into regions it saw little of in training. You can detect this immediately from the inputs alone — no labels required.
- Concept drift — the relationship P(y|X) changes. The same inputs now correspond to different outcomes: the very definition of "fraud" or "churn" evolved. The inputs can look statistically unchanged while the model's learned mapping is simply wrong now. Detecting concept drift usually needs ground-truth labels to see that accuracy fell, which often arrive with a delay.
The practical distinction: data drift you can catch from the inputs alone, right away; concept drift usually needs labels and shows up as falling accuracy. Both are reasons to retrain — but data drift is your early-warning system.
Quantifying data drift: PSI. You don't eyeball histograms in production — you compute a number. The Population Stability Index (PSI) compares a reference distribution (your training data) against the current live distribution. Bin both on the same edges, take the proportion of mass in each bin, and sum (p_cur − p_ref) · ln(p_cur / p_ref) across bins. PSI is 0 when the distributions match and grows as they diverge. A widely-used rule of thumb: PSI < 0.1 = no meaningful shift, 0.1–0.2 = moderate shift (watch it), > 0.2 = significant shift (investigate / retrain). A related, label-free check is the KS statistic — the maximum gap between the two empirical CDFs.
Alert thresholds → retraining triggers. Drift metrics feed alerts: when PSI on a key feature crosses 0.2 (or accuracy on labeled data drops below a floor), the system fires an alert and, in a mature pipeline, triggers retraining — pull fresh labeled data, retrain, re-evaluate, and ship through the same CI/CD path from lessons 12–13. That closes the ML lifecycle loop: deploy → monitor → detect drift → retrain → deploy. The cell below computes PSI and a KS-style statistic in pure NumPy and watches them rise as a distribution shifts.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
What is the difference between DATA drift and CONCEPT drift?
- A deployed model fails silently — no crash, just gradually worse predictions as the world drifts from its training data — so you must log predictions and monitor for drift.
- Data drift is a shift in the inputs P(X) (catchable from inputs alone, immediately); concept drift is a shift in the relationship P(y|X) (the same inputs map to different outcomes, usually seen via falling accuracy on labels).
- The PSI quantifies how far the live input distribution moved from training; crossing an alert threshold (e.g. PSI > 0.2) trips a retraining trigger — monitoring is the smoke detector that closes the lifecycle loop.
Production model services ship with monitoring dashboards that log predictions and track drift metrics (PSI/KS per feature, accuracy on delayed labels); a drift alert is the standard signal that triggers a retraining run.
If you remove it: Accuracy would erode invisibly with no error or crash — you'd only discover the rot when a business metric tanks or a user complains, long after the degraded predictions started.