9 · Evaluating unsupervised models
No labels? No problem — silhouette for clustering quality, ARI when you can cross-check, AUC-PR for anomaly models, and cluster stability as the ultimate sanity check.
Unsupervised evaluation: silhouette + ARI when ground truth exists, AUC-PR for labelled anomalies, sanity checks always.
Without this:
Without proper evaluation you cannot tell whether your clustering or anomaly detector is doing anything useful — you'd ship a broken model and not know it.
Supervised learning has a clear evaluation loop: split data into train/test, train on one, evaluate on the other. Unsupervised learning is harder to evaluate because there are no ground-truth labels to compare against — at least not routinely.
But "no labels" does not mean "no evaluation." We have three complementary strategies:
1. Internal metrics — quality measures that require only the data and the cluster assignments, no external labels.
- Silhouette score (−1 to +1): measures cohesion vs separation. Best general-purpose internal metric.
- Davies-Bouldin index (lower is better): ratio of within-cluster scatter to between-cluster separation.
- Calinski-Harabasz index (higher is better): ratio of between-cluster to within-cluster dispersion.
2. External metrics — compare cluster assignments to a known reference (ground truth). Only usable when labels do exist (benchmarking, validation, model comparison).
- Adjusted Rand Index (ARI) — measures agreement between two label assignments corrected for chance. Range: −1 to 1; ≈0 = random; 1 = perfect. Robust to cluster number mismatch.
- Normalized Mutual Information (NMI) — measures information shared between assignments, normalised to 0–1.
- Homogeneity — each predicted cluster contains only members of a single true class.
- Completeness — all members of a true class are in the same predicted cluster.
- V-measure — harmonic mean of homogeneity and completeness.
3. Anomaly evaluation — when you have labelled anomalies (e.g., fraud logs, flagged intrusions), evaluate with the standard precision-recall framework.
- Precision-Recall curve + Average Precision (AP): because anomalies are rare, accuracy is meaningless (99.9% accuracy by always predicting "normal"). AUC-PR is the right metric.
- AUC-ROC is a secondary check; prefer AUC-PR for class-imbalanced problems.
4. Cluster stability — refit the model multiple times with different random seeds and measure ARI between all pairs of runs. High median pairwise ARI = stable, reproducible structure. Low ARI = the model is fitting noise.
Python (in browser)
External metrics on make_blobs(centers=4) with KMeans(n=4) — ARI, NMI, homogeneity, completeness, and V-measure all near 1.0, confirming perfect recovery.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
ARI and V-measure across misspecified K — both peak at the true K=4 and degrade gracefully in both directions.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Precision-Recall curve for anomaly detection — Area under PR curve (Average Precision) far exceeds the random baseline, confirming Isolation Forest ranks anomalies above normal points.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Cluster stability: 10 KMeans runs with different seeds — pairwise ARI measures reproducibility. High median ARI confirms the K=4 partition is stable, not an artefact of random initialisation.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The complete unsupervised evaluation checklist — internal metrics, external metrics, stability, visual inspection, and domain sanity for both clustering and anomaly detection.
What is the approximate range of the Adjusted Rand Index (ARI)?
- Internal metrics (silhouette, Davies-Bouldin, Calinski-Harabasz) evaluate clustering quality with no labels required.
- External metrics (ARI, NMI, homogeneity, completeness, V-measure) compare predictions to ground-truth labels — use for benchmarking when labels exist.
- ARI ≈ 0 means random agreement; ARI = 1 means perfect; slightly negative means worse than random. It is robust to number-of-clusters mismatch.
- For anomaly detection with labelled examples, use AUC-PR (average_precision_score) — accuracy is meaningless for class-imbalanced anomaly detection.
- Cluster stability (median pairwise ARI across repeated fits) is the safest sanity check when no labels exist — stable structure means the algorithm is not fitting noise.
Production ML monitoring uses ARI drift detection to flag when cluster structure shifts over time (data drift); AUC-PR tracks anomaly model degradation in fraud systems; stability checks gate model re-deployments in automated MLOps pipelines.
If you remove it: You'd ship unsupervised models without any quality signal — no way to know if your clustering is meaningful or your anomaly detector is better than random.