Assess Models with Confusion Matrices

You've trained a classifier, and the accuracy looks great — but your users are complaining about wrong predictions, and you can't figure out why. The problem is that accuracy alone hides what's actually going wrong. A confusion matrix is the diagnostic tool that reveals exactly where your model succeeds and fails, turning vague accuracy numbers into actionable insight. In this lesson, you'll learn to use confusion matrices to assess models — interpret TP, FP, FN, TN, compute derived metrics, and use them to improve your applied AI pipelines.

The problem this lesson solves

When you evaluate a classification model with a single number like accuracy, you're looking at a summary that can be dangerously misleading. Consider a fraud detection model trained on data where 99% of transactions are legitimate. A model that predicts "not fraud" 100% of the time achieves 99% accuracy — yet it catches zero fraud. This is the classic accuracy paradox: high accuracy, useless model.

Beyond that, accuracy tells you nothing about types of errors. Is your model more likely to miss a cancer diagnosis (false negative) or to cry wolf with a false alarm (false positive)? The costs are wildly different, but accuracy treats them the same. Without a confusion matrix, you're flying blind — you can't tell whether your model is biased toward one class, which inputs it confuses, or where retraining efforts will pay off.

The pain is real: you ship a model, it underperforms in production, and you have no structured way to debug it. This lesson gives you the tool that every applied AI engineer reaches for first when assessing a classifier.

Core concept / mental model

A confusion matrix is a table that cross-tabulates your model's predictions against the actual ground truth. For binary classification, it's a 2×2 grid with four cells:

• True Positive (TP): model predicts positive, and it's actually positive (a correct hit).
• True Negative (TN): model predicts negative, and it's actually negative (a correct rejection).
• False Positive (FP): model predicts positive, but it's actually negative (a Type I error — a false alarm).
• False Negative (FN): model predicts negative, but it's actually positive (a Type II error — a missed detection).

Think of the matrix like a doctor's diagnostic test: TP is catching a disease that exists, TN is clearing a healthy patient, FP is telling a healthy person they're sick, and FN is sending a sick patient home untreated.

From these four numbers, you can derive a family of metrics: - Accuracy = (TP + TN) / (TP + TN + FP + FN) - Precision = TP / (TP + FP) — of all positive predictions, how many were correct - Recall (Sensitivity) = TP / (TP + FN) — of all actual positives, how many were caught - Specificity = TN / (TN + FP) — of all actual negatives, how many were correctly rejected - F1 Score = 2 × (Precision × Recall) / (Precision + Recall) — harmonic mean balancing precision and recall

The matrix is the ground truth; these metrics are derived interpretations. It's a mental model that forces you to think in terms of error types rather than a single aggregate score.

How it works step by step

To use a confusion matrix to assess a model, follow this logical sequence:

1. Prepare labeled test data — You need a holdout set with known ground truth. Never evaluate on training data — that only measures memorization.
2. Make predictions — Run your trained model on the test set to get predicted class labels (or probabilities, which you then threshold).
3. Compare predictions to actuals — For every sample, determine if the prediction matches reality. Tally the four counts: TP, FP, FN, TN.
4. Build the matrix — Arrange the counts in the 2×2 grid (or larger for multiclass). In Python, sklearn.metrics.confusion_matrix does this in one call.
5. Compute derived metrics — Use classification_report or manual formulas to get precision, recall, F1, etc., each illuminating a different aspect of performance.
6. Interpret the pattern — Ask questions: Are FPs high? That means the model over-predicts the positive class. Are FNs high? It's missing positives. Then decide which error type is more costly for your use case.
7. Act on findings — Adjust the decision threshold, collect more data for misclassified classes, retrain with class weights, or try a different algorithm.

Each step flows naturally into the next — the matrix is not an endpoint but a starting point for targeted model improvement.

Hands-on walkthrough

Let's put this into practice. We'll train a simple logistic regression on the breast cancer dataset, then use a confusion matrix to assess it.

First, train the model and generate predictions:

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, classification_report

# Load the dataset
X, y = load_breast_cancer(return_X_y=True)

# Split into train and test
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Train a model
model = LogisticRegression(max_iter=5000)
model.fit(X_train, y_train)

# Predict
y_pred = model.predict(X_test)

# Build the confusion matrix
cm = confusion_matrix(y_test, y_pred)
print("Confusion Matrix:")
print(cm)

Expected output (your exact numbers may vary slightly):

Confusion Matrix:
[[43  2]
 [ 4 65]]

Interpret: TP=65 (malignant correctly caught), TN=43 (benign correctly rejected), FP=2 (benign predicted as malignant), FN=4 (malignant predicted as benign). The model misses 4 actual cases of cancer — potentially critical.

Now compute the derived metrics to get the full picture:

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

print(f"Accuracy:  {accuracy_score(y_test, y_pred):.3f}")
print(f"Precision: {precision_score(y_test, y_pred):.3f}")
print(f"Recall:    {recall_score(y_test, y_pred):.3f}")
print(f"F1 Score:  {f1_score(y_test, y_pred):.3f}")

print("\nFull classification report:")
print(classification_report(y_test, y_pred, target_names=['benign', 'malignant']))

Expected output:

Accuracy:  0.947
Precision: 0.970
Recall:    0.942
F1 Score:  0.956

Full classification report:
              precision    recall  f1-score   support

     benign       0.92      0.96      0.94        45
  malignant       0.97      0.94      0.96        69

    accuracy                           0.95       114
   macro avg       0.94      0.95      0.95       114
weighted avg       0.95      0.95      0.95       114

Notice the support column shows the actual number of samples per class (45 benign, 69 malignant) — the row sums match the confusion matrix (43+2=45, 4+65=69). This cross-check validates your matrix.

Now let's build a visual confusion matrix for a clearer look:

import matplotlib.pyplot as plt
from sklearn.metrics import ConfusionMatrixDisplay

disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=['benign', 'malignant'])
disp.plot(cmap='Blues')
plt.title('Breast Cancer Confusion Matrix')
plt.show()

The plot shows the same numbers in a color-coded grid — darker cells mean higher counts. This is often easier to eyeball in a presentation or notebook.

Compare options / when to choose what

You now have multiple metrics derived from the confusion matrix. How do you choose which one matters most?

Pro tip: Always report the confusion matrix itself, not just the metrics. The raw counts let others reason about your error patterns independently.

Variation — multiclass confusion matrices: For problems with more than two classes, the matrix grows to K×K. Each row sums to the actual count for that class, and each column sums to the predicted count. You can compute precision and recall per class (macro-averaged) or weighted by support. The same principles apply — you're still tracking which classes get confused with each other.

Troubleshooting & edge cases

Problem: My matrix shows zero for one cell. This typically means your model always predicts one class — common when classes are extremely imbalanced. Fix: resample your data, use class weights, or adjust the decision threshold.

Problem: Accuracy is high but recall is terrible. This is the classic imbalanced-data trap. The model is exploiting the majority class. Solution: don't use accuracy; use precision/recall/F1 tuned for the minority class.

Problem: The matrix doesn't match my manual counts. Double-check your y_test and y_pred alignment. A common bug is predicting on the training set (leakage) or shuffling test order. Always ensure predictions come from the same holdout split.

Edge case — multi-label vs multiclass: Multi-label (each sample can belong to several classes) requires a different evaluation — for instance, averaging per label. A simple confusion matrix assumes each sample has exactly one true label.

Edge case — threshold moving: If your model outputs probabilities, the confusion matrix changes with the threshold you use to convert to labels. A lower threshold (e.g., 0.3 instead of 0.5) increases TP and FP, raising recall but lowering precision. Always note the threshold you used.

What you learned & what's next

You can now use confusion matrices to assess models, interpret the four core counts (TP, FP, FN, TN), and derive precision, recall, F1, and accuracy from them — completing both learning objectives. You've seen the mental model, walked through a hands-on breast cancer example, learned how to choose the right metric, and solved common evaluation pitfalls. This diagnostic skill is essential in any applied AI pipeline.

Next in the track, you'll likely build on this by learning to optimize your decision threshold or compare models using ROC curves — both are natural extensions of confusion-matrix thinking. With a solid grasp of evaluation, you're ready to iterate toward production-grade classifiers.

Pro tip: When you present model results to stakeholders, lead with the confusion matrix and the one or two metrics that matter for the business — not a wall of numbers.