10 · Confusion matrix, precision, recall, F1, ROC, AUC
Accuracy lies on imbalanced data — the full story needs precision, recall, F1, and AUC.
Accuracy lies on imbalanced data — precision, recall, F1, and AUC tell the full story.
Without this:
Without these, you'd ship a fraud detector that catches nothing and proudly report 99% accuracy.
A classification model produces both a predicted label and a predicted probability. Once you have a threshold (default 0.5), every sample falls into one of four cells:
The confusion matrix
| | Predicted Positive | Predicted Negative | |---|---|---| | True Positive | TP | FN | | True Negative | FP | TN |
- TP (True Positive): predicted 1, truly 1 — correct positive call.
- FP (False Positive): predicted 1, truly 0 — false alarm.
- FN (False Negative): predicted 0, truly 1 — missed positive.
- TN (True Negative): predicted 0, truly 0 — correct negative call.
From these four numbers you derive every metric:
Accuracy = (TP + TN) / (TP + FP + FN + TN)
Total fraction correct. Misleading on imbalanced data — a model that always predicts "not fraud" on a 99:1 dataset achieves 99% accuracy while catching zero frauds.
Precision = TP / (TP + FP)
"Of the samples I predicted positive, what fraction were truly positive?" Optimise precision when false positives are costly (e.g., email spam filter — you don't want legitimate mail in the spam folder).
Recall (Sensitivity) = TP / (TP + FN)
"Of all the true positives in the dataset, what fraction did I find?" Optimise recall when false negatives are costly (e.g., cancer screening — you don't want to miss a real case).
The precision-recall tradeoff: lowering the decision threshold (< 0.5) catches more positives (higher recall) but also accepts more false alarms (lower precision). Raising the threshold does the opposite.
F1 score = 2 · (precision · recall) / (precision + recall) = harmonic mean of P and R.
When precision and recall are both important and you want a single number, use F1. It is the harmonic mean — which gives lower weight to whichever metric is smaller.
Specificity = TN / (TN + FP) — the recall for the negative class.
ROC curve — plot TPR (=recall) on the y-axis vs FPR (= 1 − specificity) on the x-axis as the classification threshold varies from 1 down to 0. A random classifier gives the diagonal y = x line. A perfect classifier passes through (0, 1).
AUC (Area Under the ROC Curve) — a threshold-independent summary. AUC = 0.5 means random. AUC = 1.0 means perfect separation. AUC has a nice interpretation: it equals the probability that a randomly chosen positive sample is ranked above a randomly chosen negative sample.
PR curve (Precision-Recall curve) — plots precision vs recall as the threshold varies. More informative than ROC when positives are rare (imbalanced data).
Python (in browser)
High overall accuracy on imbalanced data is a trap. Look at the per-class precision and recall in the classification report — the minority class (fraud, disease, etc.) will show much worse numbers. This is why you never evaluate a classifier on imbalanced data with accuracy alone.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Each point on the ROC curve is a different threshold. The shaded area is the AUC. The diagonal dashed line is the random baseline. The further the curve bows toward the top-left corner, the better the model ranks positives above negatives.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
On imbalanced data (10% positives), the random baseline for precision is 0.10 — not 0.50 as in the ROC case. The PR curve exposes how bad the model really is on the minority class. A PR curve closer to the top-right is better.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The choice of beta is a business decision, not a statistical one. Ask: 'Is it worse to miss a true case (FN) or to raise a false alarm (FP)?' The answer determines beta.
A model predicts 0 for everything on a 99:1 imbalanced dataset. What are its accuracy and recall on the minority class?
- The confusion matrix (TP, FP, FN, TN) is the foundation of every classification metric.
- Precision = TP/(TP+FP) optimises for false-alarm cost; Recall = TP/(TP+FN) optimises for missed-positive cost; F1 is their harmonic mean.
- ROC-AUC is threshold-independent and measures ranking quality; PR-AUC is more honest on heavily imbalanced data.
- Never use accuracy as the sole metric on imbalanced datasets — always check per-class recall.
Every classification leaderboard (Kaggle, papers) reports AUC or F1, not accuracy. Production deployments use precision-at-K (e.g., top-100 highest-confidence predictions) for ranking problems.
If you remove it: You'd ship models that look perfect on the spreadsheet and fail on the use case.