Four counts decide everything: right about “yes”, right about “no”, and the two ways to be wrong. A Python confusion matrix is just that 2×2 tally, yet precision, recall, F1, ROC, and AUC all fall out of it by simple arithmetic. This post builds the grid by hand on ten patients, checks it against scikit-learn, then walks the ROC curve and AUC.

“The confusion matrix is misnamed. It is the one thing that removes the confusion about how your classifier actually performs.”

Sebastian Raschka, Python Machine Learning

Last Updated: July 2026 | Tested on: Python 3.14.6, scikit-learn 1.9.0 | Difficulty: Intermediate | Reading Time: 19 minutes

The model evaluation tutorial turned model quality into single numbers. This one makes those numbers visual and shows where they come from. Picture a doctor, Niranjan, screening 300 patients for an illness. For each patient he can be right two ways and wrong two ways: he flags a sick person as sick (good), clears a healthy person (good), raises a false alarm on a healthy person (annoying), or sends a sick person home untreated (dangerous). A confusion matrix is nothing more than a tally sheet for those four outcomes. Once you see it that way, the rest is counting and division.

Prerequisites

The Python Confusion Matrix: Four Numbers That Tell Everything

Every binary classifier sorts each prediction into one of four boxes. The diagram below names them. Read it as “what I predicted” versus “what was actually true”. The two green boxes are correct calls. The two red boxes are the two different ways to be wrong, and they are not equally bad, which is the whole point.

Two ideas pop straight out of the picture. Precision asks: of everyone I flagged as sick, how many really were? It is true positives divided by all the positive calls (TP over TP plus FP). Recall asks: of everyone who really was sick, how many did I catch? It is true positives divided by all the real positives (TP over TP plus FN). The F1 score is just the harmonic mean of those two, a single number that punishes you for being lopsided. Keep this diagram in mind, because every formula in the rest of the post is one of these arrows.

The Math, Worked by Hand on 10 Patients

Before any library, let us do it the slow way on a tiny set so you can check every digit yourself. Niranjan screens 10 patients. Four of them are genuinely sick (label 1), six are healthy (label 0). Here is the truth and his guesses, lined up:

📝 Ten patients, by hand

patient :  1    2    3    4    5    6    7    8    9   10
truth   :  1    1    1    1    0    0    0    0    0    0     (4 sick, 6 healthy)
guess   :  1    1    1    0    1    0    0    0    0    0
result  : TP   TP   TP   FN   FP   TN   TN   TN   TN   TN

Walk the columns one at a time. Patients 1 to 3: truth says sick, guess says sick, so each is a true positive. Patient 4: truth says sick, guess says healthy, so Niranjan missed a real case, a false negative. Patient 5: truth says healthy, guess says sick, a false alarm, a false positive. Patients 6 to 10: healthy and correctly cleared, all true negatives. Tally it up: TP = 3, FN = 1, FP = 1, TN = 5. Now the three formulas, with the real numbers plugged straight in:

📝 The three formulas with numbers filled in

Precision = TP / (TP + FP) = 3 / (3 + 1) = 3/4 = 0.75
Recall    = TP / (TP + FN) = 3 / (3 + 1) = 3/4 = 0.75
Accuracy  = (TP + TN) / 10 = (3 + 5) / 10 = 8/10 = 0.80

So Niranjan is right 80% of the time overall, but that single accuracy number hides the story. He caught 3 of the 4 sick patients (recall 0.75) and one of his four “sick” calls was a false alarm (precision 0.75). The one he missed is the scary error. That is why we never trust accuracy alone for anything where the two mistakes carry different weight.

From Scratch, Then With scikit-learn

Now let us turn that hand count into code. No imports, no magic, just a loop that drops each prediction into one of the four buckets. If you can read this loop, you understand the Python confusion matrix completely.

📄 confusion_from_scratch.py: count the four outcomes with one loop

# A tiny, hand-checkable example. 10 patients.
# 1 = sick (positive), 0 = healthy (negative).
y_true = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0]   # 4 actually sick, 6 actually healthy
y_pred = [1, 1, 1, 0, 1, 0, 0, 0, 0, 0]   # what our model guessed

# Count the four outcomes by hand, no library.
tp = fp = fn = tn = 0
for actual, predicted in zip(y_true, y_pred):
    if actual == 1 and predicted == 1:
        tp += 1          # said sick, was sick
    elif actual == 0 and predicted == 1:
        fp += 1          # said sick, was healthy (false alarm)
    elif actual == 1 and predicted == 0:
        fn += 1          # said healthy, was sick (missed it)
    else:
        tn += 1          # said healthy, was healthy

print(f"TP={tp}  FP={fp}  FN={fn}  TN={tn}")
print(f"Precision = TP/(TP+FP) = {tp}/{tp+fp} = {tp/(tp+fp):.2f}")
print(f"Recall    = TP/(TP+FN) = {tp}/{tp+fn} = {tp/(tp+fn):.2f}")
print(f"Accuracy  = (TP+TN)/10 = {tp+tn}/10 = {(tp+tn)/10:.2f}")

▶ Output

TP=3  FP=1  FN=1  TN=5
Precision = TP/(TP+FP) = 3/4 = 0.75
Recall    = TP/(TP+FN) = 3/4 = 0.75
Accuracy  = (TP+TN)/10 = 8/10 = 0.80

What happened here: the code printed exactly the numbers we worked out by hand: TP=3, FP=1, FN=1, TN=5, with precision, recall, and accuracy all matching the fractions above. The four if branches are the four boxes in the diagram, nothing more. This is the entire mechanism. Everything scikit-learn does is a faster, vectorised version of this loop.

Now the library version. confusion_matrix returns a NumPy 2×2 array. One thing trips people up: the default row and column order is sorted label order, so for labels 0 and 1 the grid reads top-left to bottom-right as TN, FP, FN, TP. We pass labels=[0, 1] to make that order explicit, then unpack with .ravel().

📄 confusion_sklearn.py: same data, same answer, one function call

from sklearn.metrics import confusion_matrix

y_true = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0]
y_pred = [1, 1, 1, 0, 1, 0, 0, 0, 0, 0]

# labels=[0, 1] forces the order so we read it the same way every time.
cm = confusion_matrix(y_true, y_pred, labels=[0, 1])
print(cm)
print()
tn, fp, fn, tp = cm.ravel()
print(f"TN={tn}  FP={fp}  FN={fn}  TP={tp}")

▶ Output

[[5 1]
 [1 3]]

TN=5  FP=1  FN=1  TP=3

What happened here: the printed grid is [[5 1] [1 3]]. Top row is the actually-healthy patients (5 cleared correctly, 1 false alarm). Bottom row is the actually-sick patients (1 missed, 3 caught). Read with .ravel() that is TN=5, FP=1, FN=1, TP=3, an exact match to our from-scratch loop. The library did not do anything mysterious; it counted the same four buckets.

The Threshold Knob

Here is the part that confuses everyone at first. A classifier does not really output “sick” or “healthy”. It outputs a probability of sick, a number between 0 and 1. The label comes from comparing that probability to a cutoff, the threshold. By default the cutoff is 0.5: probability of 0.5 or higher means “sick”. But you are free to turn that knob, and turning it reshapes the whole confusion matrix.

Think of the threshold like the sensitivity dial on a smoke detector. Crank it down and it screams at burnt toast (lots of false alarms, but it never misses a real fire). Crank it up and it stays quiet (few false alarms, but it might sleep through a small fire). Same detector, different tradeoff. Watch what happens to recall and precision when Niranjan lowers his cutoff from 0.5 to 0.35:

📄 threshold_knob.py: one set of probabilities, two cutoffs

import numpy as np
from sklearn.metrics import confusion_matrix

# Predicted probabilities of "sick" for 8 patients, and the truth.
probs  = np.array([0.95, 0.82, 0.60, 0.55, 0.45, 0.40, 0.20, 0.10])
y_true = np.array([1,    1,    0,    1,    0,    1,    0,    0])

for threshold in [0.5, 0.35]:
    preds = (probs >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(y_true, preds, labels=[0, 1]).ravel()
    recall = tp / (tp + fn)
    precision = tp / (tp + fp) if (tp + fp) else 0.0
    print(f"threshold={threshold}: TP={tp} FP={fp} FN={fn} TN={tn} "
          f"-> recall={recall:.2f}, precision={precision:.2f}")

▶ Output

threshold=0.5: TP=3 FP=1 FN=1 TN=3 -> recall=0.75, precision=0.75
threshold=0.35: TP=4 FP=2 FN=0 TN=2 -> recall=1.00, precision=0.67

What happened here: at the 0.5 cutoff the model caught 3 of 4 sick patients (recall 0.75) with one false alarm (precision 0.75). Drop the cutoff to 0.35 and the patient with probability 0.40 now counts as “sick”, so recall jumps to a perfect 1.00 (zero missed cases) but a second healthy patient gets flagged, dragging precision down to 0.67. Nothing about the model changed, only the knob. This push-pull between catching more real cases and raising more false alarms is the single design decision behind every classifier you ship. A cancer screen wants high recall even at the cost of precision; a spam filter that dumps a real invoice into junk wants the opposite.

ROC Curves and AUC: Performance Across All Thresholds

Picking one threshold gives you one Python confusion matrix. But what if you slide the threshold across every possible value, from 1.0 all the way down to 0.0, and plot the result? That is the ROC (Receiver Operating Characteristic) curve. Think of it like testing an airport metal detector at every sensitivity setting at once: at each setting you note how many real weapons it catches versus how often it beeps at a harmless belt buckle, then you chart that whole sweep on a single graph.

The ROC curve charts the true positive rate (recall) on the vertical axis against the false positive rate on the horizontal axis, one point per threshold. The diagram below shows the intuition: a high threshold sits in the bottom-left corner (catch little, false-alarm little), and as you lower it you climb up and to the right.

The left column traces the threshold sliding from high to low: you start with low recall and few false positives, and as you move the cutoff down both climb. The right column is the scorecard for the whole curve. The AUC (area under the curve) collapses the entire ROC line into one number from 0.5 to 1.0. AUC of 0.5 is a coin flip; the curve sits on the diagonal and the model has learned nothing. AUC of 1.0 is a flawless ranker. Most useful real models land somewhere in between, often 0.8 to 0.95.

Here is the cleanest way to think about AUC, and it has nothing to do with area at first glance: AUC is the probability that the model gives a randomly picked sick patient a higher “sick” score than a randomly picked healthy patient. An AUC of 0.92 means that if you grab one sick person and one healthy person at random, 92 times out of 100 the model ranks the sick one higher. That is why AUC does not care about your chosen threshold at all; it measures ranking quality, not where you happened to draw the line.

Let us compute it for real on a 1000-sample dataset with two models. We seed everything (random_state=42) so your numbers match these exactly.

📄 roc_auc_demo.py: AUC for two models, plus the first ROC points

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, roc_curve

X, y = make_classification(n_samples=1000, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42)

models = {
    "Logistic Regression": LogisticRegression(random_state=42),
    "Random Forest": RandomForestClassifier(n_estimators=100, random_state=42),
}

print("Model                  | AUC")
print("-----------------------|------")
for name, model in models.items():
    model.fit(X_train, y_train)
    probas = model.predict_proba(X_test)[:, 1]
    auc = roc_auc_score(y_test, probas)
    print(f"{name:<23}| {auc:.4f}")

# Show the first few points of the Logistic Regression ROC curve.
lr = models["Logistic Regression"]
fpr, tpr, thresholds = roc_curve(y_test, lr.predict_proba(X_test)[:, 1])
print()
print("First ROC points (Logistic Regression):")
print("  FPR     TPR     threshold")
for i in range(4):
    t = thresholds[i]
    t_str = "  inf" if np.isinf(t) else f"{t:.3f}"
    print(f"  {fpr[i]:.3f}   {tpr[i]:.3f}   {t_str}")

▶ Output

Model                  | AUC
-----------------------|------
Logistic Regression    | 0.9142
Random Forest          | 0.9244

First ROC points (Logistic Regression):
  FPR     TPR     threshold
  0.000   0.000     inf
  0.000   0.006   1.000
  0.000   0.387   0.961
  0.007   0.387   0.961

What happened here: both models score in the low 0.91 to 0.92 range, solidly “good but not perfect”. Random Forest edges out Logistic Regression (0.9244 versus 0.9142), so it ranks positives above negatives slightly more often. The ROC points show how the curve is built: roc_curve walks the thresholds from highest to lowest. The first point is the conventional start at (0, 0) with an infinite threshold (nobody is flagged).

As the threshold drops to 0.961, the true positive rate jumps to 0.387 while the false positive rate barely moves, which is exactly the steep climb up the left edge you want to see. The number you report to stakeholders is the AUC; the curve is how you choose the operating threshold afterward.

In a notebook you would draw the curve with RocCurveDisplay.from_estimator(model, X_test, y_test), which produces a chart with the diagonal “random” baseline and the AUC printed in the legend. The shape tells you the story at a glance: the closer the curve hugs the top-left corner, the better the model, and the bigger the area underneath.

When This Wins and When It Loses

AUC has one famous blind spot: heavy class imbalance. Because the false positive rate has the giant true-negative count in its denominator, a model can rack up an impressive ROC AUC while still being almost useless at finding the rare positives you actually care about. Fraud, rare diseases, and defect detection all live here. The honest score for those cases is PR AUC (precision-recall AUC, also called average precision), which ignores the easy negatives entirely. Watch the gap on a dataset that is only 5% positive:

📄 roc_vs_pr.py: same model, two very different verdicts

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, average_precision_score

# Heavily imbalanced: only 5% positives.
X, y = make_classification(n_samples=5000, weights=[0.95, 0.05],
                           random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42)

model = LogisticRegression(random_state=42, max_iter=1000).fit(X_train, y_train)
probas = model.predict_proba(X_test)[:, 1]

roc_auc = roc_auc_score(y_test, probas)
pr_auc = average_precision_score(y_test, probas)

print(f"Positives in test set: {y_test.sum()} out of {len(y_test)} "
      f"({100 * y_test.mean():.1f}%)")
print(f"ROC AUC = {roc_auc:.3f}  (looks great)")
print(f"PR  AUC = {pr_auc:.3f}  (the honest score on the rare class)")

▶ Output

Positives in test set: 78 out of 1500 (5.2%)
ROC AUC = 0.898  (looks great)
PR  AUC = 0.543  (the honest score on the rare class)

What happened here: the ROC AUC of 0.898 looks like a strong model. But only 5.2% of the test set is positive, and the PR AUC tells the real story: 0.543, barely better than a coin flip on the class that matters. The same predictions, judged two ways, give two opposite impressions. The lesson for this section: use ROC AUC when your classes are roughly balanced and both errors matter; switch to PR AUC the moment the positive class is rare. A Python confusion matrix is great for one fixed threshold, AUC summarises all thresholds, and PR AUC keeps you honest when the negatives vastly outnumber the positives.

Common Mistakes

Mistake 1: Reading the confusion matrix in the wrong orientation

🚫 Wrong

cm = confusion_matrix(y_true, y_pred)
tp = cm[0][0]    # WRONG: cm[0][0] is the true NEGATIVE for labels 0/1

✅ Correct

cm = confusion_matrix(y_true, y_pred, labels=[0, 1])
tn, fp, fn, tp = cm.ravel()   # unambiguous: rows=actual, cols=predicted

Why: scikit-learn sorts labels, so for a 0/1 problem cm[0][0] is the true negative, not the true positive. Hard-coding indices is how people quietly swap precision and recall in a report. Pass labels=[0, 1] and unpack with .ravel() so the four names are never in doubt.

Mistake 2: Trusting AUC on heavily imbalanced data

🚫 Wrong

# Fraud is 0.5% of rows. "0.97 AUC, ship it!"
auc = roc_auc_score(y_test, probas)   # inflated by the huge TN count

✅ Correct

# Judge the rare positive class with precision-recall AUC.
pr_auc = average_precision_score(y_test, probas)

Why: false positive rate divides by the true-negative count, which is enormous when negatives dominate. That makes ROC AUC look rosy even when precision on the positive class is poor. As the section above showed, a 0.898 ROC AUC hid a 0.543 PR AUC. On rare-event problems, report PR AUC.

Conclusion

You started with a Python confusion matrix, four counts in a 2×2 grid, and ended up holding every classification metric that matters. You tallied TP, FP, FN, and TN by hand on ten patients, reproduced them with a plain Python loop, and matched them against scikit-learn’s confusion_matrix. From there you turned the threshold knob and watched precision and recall trade places, walked the ROC curve as a sweep across every threshold, and learned that AUC is really just the probability your model ranks a true positive above a true negative. Finally you saw AUC’s blind spot on imbalanced data and why PR AUC keeps you honest when the positive class is rare.

Next up are decision trees, where you move from scoring a model to building one you can actually read, following how it splits the data to reach each prediction. For the full learning path, from first Python line to production ML, see the Python + AI/ML tutorial series home.

Frequently Asked Questions

Interview Questions on Confusion Matrix, ROC, and AUC

How interviewers actually probe this topic: real scenarios, with answers you can say out loud.

Q: In a 0/1 problem, what does the top-left cell of scikit-learn’s confusion_matrix contain, and why does it trip people up?

scikit-learn sorts labels and puts true labels on rows, predicted labels on columns, so for labels 0 and 1 the top-left cell cm[0][0] is the true negative, not the true positive. People assume the top-left is always TP and end up swapping precision and recall. Pass labels=[0, 1] and unpack with tn, fp, fn, tp = cm.ravel() so the four names are never in doubt.

Q: Explain AUC in one sentence without mentioning the word “area”.

AUC is the probability that the model assigns a higher positive score to a randomly chosen positive example than to a randomly chosen negative example. An AUC of 0.90 means that 90 times out of 100 a random positive outranks a random negative. Because it only depends on the ranking of scores, AUC is independent of any single classification threshold.

Q: How does lowering the classification threshold change precision and recall?

Lowering the threshold flags more examples as positive, so recall (true positive rate) goes up or stays the same because you miss fewer real positives. But you also catch more false alarms, so precision usually drops. It is a tradeoff dictated by the cost of each error: a cancer screen lowers the threshold to protect recall, while a spam filter raises it to protect precision.

Q: Why can accuracy be misleading and how does the confusion matrix expose the problem?

On imbalanced data, a model that always predicts the majority class can score very high accuracy while catching none of the rare cases. The confusion matrix exposes this instantly: you see a large true-negative count, a large false-negative count, and almost no true positives. Recall and PR AUC reveal the failure that a single accuracy number hides.

Q: Scenario: a teammate reports a 0.97 ROC AUC on a fraud model and wants to ship it. What do you check first?

Check the class balance first. Fraud datasets are often under 1% positive, and the false positive rate divides by the enormous true-negative count, which inflates ROC AUC even when the model is weak on actual fraud. Ask for the PR AUC (average precision) and the precision and recall at the intended operating threshold. If PR AUC is low, say 0.4, the model is nowhere near shippable regardless of the impressive ROC number.

Q: Scenario: your model catches almost every positive but analysts complain about too many false alarms. Which knob do you turn and what do you monitor?

High recall with too many false alarms means your threshold is set too low, so raise it. As you raise it, precision climbs and false positives fall, but watch recall so you do not start missing the real positives you care about. Use the ROC or precision-recall curve to pick a threshold that meets the false-alarm budget while holding recall at an acceptable level, rather than guessing.

Q: How would you extend ROC AUC to a multi-class problem?

Use the one-vs-rest approach: compute one ROC curve per class, treating that class as positive and all others as negative, then aggregate the AUC scores with a macro or weighted average. In scikit-learn you call roc_auc_score(y_true, y_score, multi_class='ovr'). Macro averaging weights each class equally, which matters when some classes are rare.

Series: Python + AI/ML Cookbook, Part 5: Machine Learning

Further reading: scikit-learn documentation is the authoritative source on this.

Related Posts

Previous: ML: Model Evaluation Metrics (Accuracy, Precision, Recall, F1)

Next: ML: Decision Trees in Python: Splitting, Pruning, Visualization

Series Home: Python + AI/ML Tutorial Series