2 · Detecting and handling outliers
An outlier breaks the assumed distribution — keep, remove, or transform, but never ignore.
An outlier is a value that breaks the assumed distribution — keep, remove, or transform, but never ignore.
Without this:
Without outlier handling, your mean is meaningless, your standard deviation explodes, and your model fits noise instead of signal.
An outlier is a data point that lies far from the bulk of the distribution. It might be a genuine rare event (a CEO salary in a dataset of employees), a data entry error (age = 999), or a measurement glitch. The key question is not "is this value extreme?" but "why is it extreme — and what should I do about it?"
Three detection methods dominate in practice:
- Z-score: how many standard deviations from the mean? Flag
|z| > 3. Works well when data is approximately normal. - IQR rule: define fences at Q1 − 1.5·IQR and Q3 + 1.5·IQR. More robust than z-score for skewed data.
- Modified Z-score (MAD): replace the mean with the median and the std with the Median Absolute Deviation. Even more robust for heavily skewed or small samples.
And three responses:
- Remove — if you're confident it's a data entry error.
- Cap / Winsorize — clamp extreme values to a percentile (e.g. 5th–95th). Preserves row count.
- Transform — apply log or sqrt to compress the right tail. Works when the column is right-skewed.
Python (in browser)
Z-score flags outliers relative to the mean ± 3 std. Here the CEO salary gets z ≈ 2.4 — the z-score method barely catches it because the outlier inflates the std too.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The IQR rule is resistant to the outlier itself because it's based on quartiles (rank-based), not the mean.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Boxplot shows the IQR box and whiskers; scatter lets you see exactly which observations are extreme.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Winsorizing preserves every row while limiting the influence of extremes — the std drops dramatically when the $1M salary is capped.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Rule of thumb: transform first (if skewed), then winsorize residual extremes, then consider removal only for confirmed errors.
A column has skew=3 and z-score flags 10% of rows as outliers. What should you suspect?
Moments of a distribution — mean (1st moment), variance (2nd), skewness (3rd). The skewness formula explains why heavily skewed data breaks z-score-based outlier detection.
- Use **z-score** (|z| > 3) for approximately normal data; **IQR rule** for skewed or small samples. Both flag extremes — z-score is distorted by the outlier itself.
- Three responses: **remove** (confirmed errors), **winsorize** (cap to percentile — preserves rows), **transform** (log/sqrt for skewed columns).
- Tree models tolerate outliers; linear and distance-based models are severely affected — choose your outlier strategy based on the downstream model.
- Use `np.log1p` for zero-inclusive columns; `PowerTransformer(method='yeo-johnson')` for any real-valued column with a skewed distribution.
Anomaly detection (Isolation Forest, LOF) explicitly hunts outliers. Robust regression uses Huber loss to reduce outlier influence. Pre-processing in sklearn often combines `RobustScaler` (uses median + IQR) for outlier-resistant scaling.
If you remove it: A single mistyped salary of 1,000,000 in a hundred-row dataset will swamp the mean and ruin a linear regression.