Measure Classification Accuracy

You've built a classifier that correctly labels incoming requests 92% of the time — impressive, right? Not so fast. If your system flags 100 alerts and 60 of them are false alarms, your precision just crashed to 40%, and your engineering team now distrusts every alert. Accuracy alone is a dangerous metric for imbalanced data, and that's exactly why measuring classification accuracy and precision properly is the difference between a model that looks good in a notebook and one that earns trust in production.

The problem this lesson solves

When you train a classification model, the obvious question is simple: "How good is it?" The obvious answer — accuracy = correct predictions / total predictions — hides a trap. Consider a fraud-detection model: if 99.7% of transactions are legitimate, a model that predicts "not fraud" for every single transaction achieves 99.7% accuracy. It's useless, but the number says otherwise.

This is the core pain: a single accuracy number can mislead you into deploying a broken system. Production ML isn't about how often your model is right in the aggregate — it's about which mistakes it makes and how costly those mistakes are. Failing to distinguish between a false positive (false alarm) and a false negative (missed fraud) leads to wasted engineering time, eroded stakeholder trust, and decisions made on gut feeling rather than evidence.

By the end of this lesson, you'll be able to measure classification accuracy and precision with confidence, know exactly when to use each, and build a small evaluation report that surfaces the truth about your model.

Core concept / mental model

Think of a classification model as a gatekeeper with two jobs: let good people in and keep bad people out. Accuracy asks: of 100 people who showed up, how many did the gatekeeper handle correctly? Precision asks a sharper question: of the people the gatekeeper let in, how many were actually supposed to be there?

Here's the vocabulary you need — the confusion matrix is a 2x2 grid of outcomes:

Two formulas sit at the center of everything:

• Accuracy = (TP + TN) / (TP + TN + FP + FN) — the share of all predictions that were correct.
• Precision = TP / (TP + FP) — the share of positive predictions that were actually correct.

Pro tip: Precision is your answer to "when the model says yes, should I believe it?" If a false alarm costs your team hours of manual review, precision matters more than accuracy.

You'll also encounter recall (True Positive Rate = TP / (TP + FN)) in most discussions. Where precision punishes false alarms, recall punishes missed positives. In this lesson, we keep the focus on accuracy and precision as your starting pair, but you'll see the symmetry.

How it works step by step

Computing accuracy and precision by hand is mechanical once you have the four numbers from the confusion matrix. Here's the workflow:

1. Collect predictions — Run your model on a labeled test set you haven't trained on. If you reuse the training data, you're measuring memorization, not learning.
2. Compare against ground truth — Walk through each example and count: did your model predict 1 when the label is 1 (TP), predict 1 when the label is 0 (FP), predict 0 when the label is 1 (FN), or predict 0 when the label is 0 (TN)?
3. Compute accuracy — Take the two correct counts (TP + TN) and divide by everything.
4. Compute precision — Take the truly positive predictions (TP) and divide by all positive predictions (TP + FP).
5. Sanity-check the context — A 95% precision on a spam filter means 19 of every 20 flagged emails are actually spam. A 95% precision on a medical screening tool might still mean thousands of needless biopsies.

The cause-and-effect chain is simple: your classifier's decision threshold sets the balance. Lower the threshold to catch more positives, and you'll boost recall but let in more false positives — precision drops. Raise the threshold, and precision climbs while you start missing true positives. The metric you optimize is a business decision, not a math one.

Hands-on walkthrough

Let's make it concrete with Python. We'll define a small dataset, compute the metrics from first principles, and then confirm with scikit-learn.

Step 1: Compute from first principles

# Ground truth and predictions: 1 = positive, 0 = negative
y_true = [1, 0, 1, 1, 0, 1, 0, 0, 1, 0]
y_pred = [1, 0, 1, 0, 0, 1, 1, 0, 1, 0]

# Build the confusion matrix counts
tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
tn = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 0)
fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)

accuracy = (tp + tn) / (tp + tn + fp + fn)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0

print(f"TP={tp} TN={tn} FP={fp} FN={fn}")
print(f"Accuracy: {accuracy:.2f}")
print(f"Precision: {precision:.2f}")

TP=4 TN=4 FP=1 FN=1
Accuracy: 0.80
Precision: 0.80

Here the numbers agree because the dataset is balanced (5 positives, 5 negatives). That's the case where accuracy isn't misleading — but it's the exception, not the rule.

Step 2: The imbalanced case that breaks accuracy

Now let's simulate the fraud-detection scenario: 997 legitimate, 3 fraudulent.

import numpy as np

y_true = np.array([0]*997 + [1]*3)
# A naive model: always predicts "not fraud"
y_pred = np.zeros(1000, dtype=int)

tp = int(((y_true == 1) & (y_pred == 1)).sum())
tn = int(((y_true == 0) & (y_pred == 0)).sum())
fp = int(((y_true == 0) & (y_pred == 1)).sum())
fn = int(((y_true == 1) & (y_pred == 0)).sum())

accuracy = (tp + tn) / (tp + tn + fp + fn)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0

print(f"Accuracy: {accuracy:.4f}")
print(f"Precision: {precision:.4f} (no positive predictions, so 0 by definition)")

Accuracy: 0.9970
Precision: 0.0000 (no positive predictions, so 0 by definition)

A 99.7% accurate model catches nothing. This is the moment where measuring classification accuracy and precision clearly — together — saves you from a catastrophic deployment.

Step 3: Confirm with scikit-learn

For production code, rely on battle-tested libraries:

from sklearn.metrics import accuracy_score, precision_score, confusion_matrix

y_true = [1, 0, 1, 1, 0, 1, 0, 0, 1, 0]
y_pred = [1, 0, 1, 0, 0, 1, 1, 0, 1, 0]

print("Confusion matrix (rows=true, cols=pred):")
print(confusion_matrix(y_true, y_pred))
print(f"Accuracy: {accuracy_score(y_true, y_pred):.2f}")
print(f"Precision: {precision_score(y_true, y_pred):.2f}")

Confusion matrix (rows=true, cols=pred):
[[4 1]
 [1 4]]
Accuracy: 0.80
Precision: 0.80

The confusion_matrix output reads as [[TN, FP], [FN, TP]] — keep that ordering in mind or you'll invert your counts.

Compare options / when to choose what

Accuracy and precision are not rivals; they answer different questions. Here's when to lean on each:

Decision rule: If your classes are roughly balanced, start with accuracy. If you're flagging problems for human review, prioritize precision. If missing a positive has severe consequences, track recall alongside precision — and consider the F1 score (harmonic mean of precision and recall) as a single summary metric to report.

Troubleshooting & edge cases

The model never predicts the positive class. If tp + fp == 0, precision is undefined — formulas crash with a ZeroDivisionError or a NaN. Always guard the denominator: in sklearn, zero_division=0 parameter lets you control what gets returned. In your own code, use the if (tp + fp) > 0 else 0.0 pattern from the examples above.

Inverted confusion matrix. Different libraries and tutorials order the matrix differently — some use (actually-positive, predicted-positive) as the top-left. Always print the matrix with labels= set explicitly, or verify against a few known cases before trusting your numbers.

Leaked labels. If you computed precision on the training set, you're optimistic by definition. A model's metrics always look better in-sample. Use train_test_split or a holdout validation set before measuring — and re-verify on data the model has truly never seen.

Warning signs in your output:

• Accuracy is high (above 90%) but precision is low (below 60%) — you're dominating with negatives; dig into why.
• Precision fluctuates wildly across runs — your test set is too small, or your training split is unstable. Increase your validation set size.
• Precision is suspiciously identical to accuracy — your dataset is likely perfectly balanced; confirm your counts before drawing conclusions.

What you learned & what's next

You now understand that measuring classification accuracy and precision is a two-part story: compute the numbers confidently with a confusion matrix, then interpret them in context. You can explain why a 99.7% accurate fraud detector can be worthless, you can compute both metrics from first principles and with sklearn, and you know when to prefer precision over accuracy — and why recall and F1 are the next pieces of the puzzle.

The practical skills you've built here — reading a confusion matrix, guarding against division-by-zero, and sanity-checking metrics against a baseline — transfer directly to evaluating any structured pipeline, including the LLM output validators and retrieval systems you'll build later in this track. Next up, you'll extend this evaluation mindset to regression metrics, where predicting a continuous value changes the game entirely. You'll use the same discipline — never trust a single number — but with MAE, RMSE, and R² as your new tools.