Evaluate Classification with Confusion Matrix

You've trained a classifier, it prints an accuracy score of 0.87, and you're ready to ship it to production. But wait — what if your model is missing every single fraud case while acing the 99% of normal transactions? A single accuracy number hides exactly that kind of failure. That's the pain this lesson solves: evaluating classification with a confusion matrix gives you the full breakdown of what your model gets right and wrong, so you can see the failures behind the score and make a confident, informed decision.

The problem this lesson solves

Accuracy alone is a dangerous metric for classification. Consider a model that flags fraudulent credit card transactions. Fraud is rare — maybe 1 in 1,000 transactions. A model that always predicts 'normal' achieves 99.9% accuracy and is completely useless. It never catches a single fraud.

The problem: a single number (accuracy) collapses all outcomes into one result. It cannot tell you:

• What kinds of errors your model makes (false alarms vs. missed catches)
• Whether your model is biased toward the majority class (e.g., always predicting 'normal')
• Which metric matters for your business (e.g., fraud detection cares about catching fraud, not about being right on normal cases)

Evaluate classification with confusion matrix solves this by decomposing predictions into four categories: true positives, true negatives, false positives, and false negatives. From those four numbers, you can derive precision, recall, and F1-score — metrics that expose the model's real strengths and weaknesses.

In this lesson, you'll learn to build and interpret a confusion matrix in Python, compute derived metrics, and use them to diagnose model flaws — the essential skill for any real-world classifier evaluation.

Core concept / mental model

Think of a confusion matrix as a 2×2 scoreboard for a binary classifier. On one axis you have the actual class (ground truth); on the other, the predicted class. Every prediction lands in one of four cells:

Let's connect the names to intuition:

• True Positive (TP) — you predicted 'yes', and the actual was 'yes'. A correct catch (e.g., flagged fraud that really was fraud).
• True Negative (TN) — you predicted 'no', and the actual was 'no'. A correct rejection (e.g., approved a normal transaction).
• False Positive (FP) — you predicted 'yes', but the actual was 'no'. A false alarm (type I error).
• False Negative (FN) — you predicted 'no', but the actual was 'yes'. A missed catch (type II error).

The matrix itself labels rows and columns with the class names, so you can see which class is which. It's called a confusion matrix because it shows how the model confuses one class for another.

From these four cells, you can define the most-common derived metrics:

• Accuracy = (TP + TN) / (TP + TN + FP + FN) — fraction of all correct predictions.
• Precision = TP / (TP + FP) — of all positive predictions, how many were correct? (Focus on false alarms.)
• Recall (Sensitivity) = TP / (TP + FN) — of all actual positives, how many did we catch? (Focus on missed catches.)
• F1-score = 2 × (Precision × Recall) / (Precision + Recall) — harmonic mean balancing both.

Why the matrix is the foundation

Every classification metric — precision, recall, F1, specificity, ROC-AUC — ultimately comes from these four cells. Once you have the confusion matrix, you can compute any of them. It's the common language for model evaluation in data science.

Pro tip: Always look at the confusion matrix first, before trusting any single metric. It's the ground truth of how your model behaves.

How it works step by step

Building and interpreting a confusion matrix follows a repeatable process:

1. Split your data into training and test sets (never evaluate on training data — you'll get unrealistic optimism).
2. Train your classifier on the training set.
3. Predict on the test set.
4. Build the confusion matrix by comparing y_true (actual test labels) with y_pred (model predictions).
5. Derive metrics — accuracy, precision, recall, F1 — from the matrix cells.
6. Interpret the matrix: look at off-diagonal cells (FP and FN) to identify failure modes.

Cause and effect

• If your FN is high, your model fails to catch positive cases — bad for fraud detection or disease screening. You need higher recall.
• If your FP is high, your model fires too many false alarms — annoying in spam filters or alert systems. You need higher precision.
• A balanced F1 might hide both being mediocre — always check the raw matrix.

Hands-on walkthrough

Let's implement a complete example: a logistic regression classifier on a synthetic dataset, evaluated with a confusion matrix. We'll use scikit-learn and pandas.

Step 1: Setup and data

# Import libraries
import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, classification_report

# Create a synthetic binary classification dataset (1000 samples, 5 features)
X, y = make_classification(n_samples=1000, n_features=5, n_informative=4,
                           n_redundant=0, random_state=42)

# Split into 70% train, 30% test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Train a logistic regression classifier
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)

Step 2: Predict and build the confusion matrix

# Predict on test set
y_pred = model.predict(X_test)

# Build the confusion matrix
cm = confusion_matrix(y_test, y_pred)
print("Confusion matrix:")
print(cm)

Expected output (may vary slightly):

Confusion matrix:
[[115  31]
 [ 20 134]]

Here, the layout by default is: [[TN, FP], [FN, TP]] in scikit-learn. So TN=115, FP=31, FN=20, TP=134.

Step 3: Derive metrics from the matrix

# Extract cells
TN, FP, FN, TP = cm.ravel()

# Compute metrics manually
total = TN + FP + FN + TP
accuracy = (TP + TN) / total
precision = TP / (TP + FP)
recall = TP / (TP + FN)
f1 = 2 * precision * recall / (precision + recall)

print(f"Accuracy:  {accuracy:.3f}")
print(f"Precision: {precision:.3f}")
print(f"Recall:    {recall:.3f}")
print(f"F1-score:  {f1:.3f}")

Expected output:

Accuracy:  0.830
Precision: 0.812
Recall:    0.870
F1-score:  0.840

The classification_report from scikit-learn gives you all these at once:

from sklearn.metrics import classification_report

print(classification_report(y_test, y_pred, target_names=['negative', 'positive']))

Expected output (illustrative):

              precision    recall  f1-score   support

    negative       0.85      0.79      0.82       146
    positive       0.81      0.87      0.84       154

    accuracy                           0.83       300
   macro avg       0.83      0.83      0.83       300
weighted avg       0.83      0.83      0.83       300

Step 4: Visualizing the matrix (optional but helpful)

import matplotlib.pyplot as plt
import seaborn as sns

plt.figure(figsize=(6, 5))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues',
            xticklabels=['Negative', 'Positive'],
            yticklabels=['Negative', 'Positive'])
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.title('Confusion Matrix')
plt.show()

This visual shows the matrix with counts in each cell, making it easy to spot where the model confuses classes.

Compare options / when to choose what

Not all metrics are created equal. The right one depends on the cost of mistakes.

Precision vs. Recall trade-off: You can't have both perfectly. Increasing recall (catching more positives) often increases FP, lowering precision. The F1-score finds a harmonic balance.

When to use each:

• Use accuracy only when classes are balanced and errors cost the same.
• Use precision when FP is costly.
• Use recall when FN is costly.
• Use F1 when you need a single number balancing both.

Pro tip: Always report the confusion matrix along with any metric. It gives context that a single number can't convey.

Troubleshooting & edge cases

1. Matrix appears reversed

If you're used to the layout [[TP, FP], [FN, TN]] (from some textbooks), you'll misread scikit-learn's default [[TN, FP], [FN, TP]]. Always check the labels parameter or use cm.ravel() carefully. Use labels=[1, 0] if you want the matrix in a different order.

cm = confusion_matrix(y_test, y_pred, labels=[1, 0])
# Now rows/cols are [positive, negative]

2. Wrong ravel() indices

Extracting cells manually from ravel() assumes the default order [TN, FP, FN, TP]. If you reordered labels, the indices change. Prefer using cm[0,0] etc. with explicit index labels, or use classification_report.

3. Class imbalance hides poor performance

If one class dominates (e.g., 99% negative), accuracy might be 0.99 while recall on the positive class is 0. A confusion matrix exposes this: you'll see a large TN but a tiny TP. Always check per-class metrics.

4. make_classification includes n_redundant

If you don't set random_state, results vary — fine for learning but confusing when debugging. Always set it for reproducibility.

5. max_iter warning in LogisticRegression

If you see a ConvergenceWarning, increase max_iter (e.g., max_iter=1000). It doesn't affect the matrix concept but prevents warning noise.

6. Multi-class cases

The confusion matrix extends to n classes as an n×n matrix. The same logic applies: diagonals are correct predictions, off-diagonals are errors. In that case, you evaluate precision/recall per class or use macro/micro averaging.

What you learned & what's next

You now understand why accuracy alone is insufficient, how to evaluate classification with confusion matrix in Python, and how to derive precision, recall, and F1 from its cells. You can identify whether your model is missing positives or causing false alarms, and choose the right metric for your business case.

Key takeaways from this lesson:

• A confusion matrix breaks predictions into TP, TN, FP, FN.
• Accuracy hides class-imbalance failures.
• Precision and recall expose specific error types.
• The matrix is the basis for many classification metrics.
• Use scikit-learn's confusion_matrix() and classification_report() for quick evaluation.

What's next: In the next lesson, you'll dive into ROC curves and AUC to evaluate classifier performance across all thresholds, building directly on the concepts you just mastered. You'll learn how to visualize the trade-off between true positive rate and false positive rate — the natural complement to the confusion matrix.

Keep practicing: re-run the example with different random_state values or flip the class weights to see how the matrix shifts. The more you inspect real predictions, the better your intuition becomes.