1 · Handling missing values
NaNs are not zeros — they're absent information, and how you fill them changes what your model learns.
Missing values are not zeros — they're absent information, and how you fill them changes what your model learns.
Without this:
Without explicit handling, NaNs propagate through every computation (silent NaNs in losses, broken metrics) and most models simply crash.
Real-world data is rarely complete. Sensor failures, optional survey fields, database joins that don't match, users who skipped a form step — all of them leave missing values behind. In pandas these appear as NaN (from NumPy), None (Python's null), or pd.NA (the newer nullable-integer sentinel).
Before fitting any model you must decide what to do with those holes. The four broad strategies are:
- Drop rows or columns that have missing values.
- Fill with a constant (e.g. 0, "Unknown", the string "Missing").
- Fill with a statistic — mean, median (numeric), or mode (categorical).
- Model-based imputation — predict the missing value from other features (KNN, iterative/MICE).
Choosing the wrong strategy is surprisingly easy and surprisingly costly. This lesson gives you the vocabulary and the code to make that choice deliberately.
Python (in browser)
`df.isna().sum()` counts absolute missingness; `df.isna().mean() * 100` expresses it as a percentage of rows — always run both at the start of an EDA.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Each strategy produces a different result. `fillna(median)` is usually the safest default for numeric features with < 20% missing.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`KNNImputer` finds the k nearest rows (ignoring the missing column) and averages their values — better than median when features are correlated.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
MCAR is safe; MAR is manageable; MNAR requires extra care — always ask WHY data is missing, not just WHERE.
If 5% of each of 10 features is randomly missing (independently), roughly what fraction of rows have at least one missing value?
- Detect missingness with `df.isna().sum()` (count) and `df.isna().mean() * 100` (percent) — always audit before imputing.
- Three patterns: **MCAR** (safe to drop/impute), **MAR** (impute with conditioning), **MNAR** (the dangerous one — missingness correlates with the missing value itself).
- Four strategies: drop, fill constant, fill statistic (median for numeric, mode for categorical), model-based (KNN/iterative). `KNNImputer` is context-aware but slower.
- For categorical features, consider creating a `'Missing'` sentinel category instead of imputing — missingness itself can be a signal.
Every ML pipeline starts with a `df.isna().sum()` audit. sklearn `Pipeline` instances need an `Imputer` step before any scaler/estimator. Tree models (XGBoost, LightGBM) handle NaN natively — most other models don't.
If you remove it: Your `model.fit(X, y)` crashes with 'Input contains NaN' — the failure mode every junior ML engineer hits in week one.