Handling Imbalanced Datasets

You've built a model that scores 97% accuracy, then you check the confusion matrix and realize it predicts the majority class for everything — your 3% of actual positive cases are completely invisible. This is the silent trap of imbalanced datasets, and it's more common than you think: fraud detection, rare disease diagnosis, churn prediction, and manufacturing defect detection all suffer from it. In this lesson, you'll stop relying on deceptive accuracy and learn how to handle imbalanced datasets with resampling, class weights, and proper evaluation metrics — so your model actually learns from the minority class, not just the majority.

The problem this lesson solves

Imagine you're building a fraud detection system for credit card transactions. Out of 100,000 transactions, only 200 are fraudulent. If you build a simple logistic regression without any adjustments, the model quickly figures out it can get 99.8% accuracy by predicting "not fraud" for every single transaction. It looks brilliant on paper, but it's completely useless — you catch zero fraud.

This is the accuracy paradox: in an imbalanced dataset, accuracy becomes a misleading metric because the majority class dominates the prediction. The real problem isn't just the class distribution — it's that most machine learning algorithms are designed to minimize overall error, which naturally biases them toward the majority class.

Unless you actively handle imbalanced datasets, your model will learn a trivial decision boundary that ignores the minority class entirely, no matter how complex your algorithm is.

Core concept / mental model

Think of training a model as teaching a student to spot rare birds in a forest. If you show them 1,000 sparrows and only 10 kingfishers, they'll learn to say "not a kingfisher" for everything and still be 99% correct. To fix this, you have two main strategies:

• Change the data so the student sees more kingfishers (resampling).
• Change the grading so missing a kingfisher costs more than mislabeling a sparrow (class weights or cost-sensitive learning).

In machine learning terms, you have two categories of techniques:

1. Data-level methods: change the class distribution before training. - Oversampling: duplicate or synthesize samples from the minority class (e.g., SMOTE). - Undersampling: remove samples from the majority class.

2. Algorithm-level methods: modify the learning algorithm to penalize mistakes on the minority class more heavily. - Class weights in logistic regression, random forests, or SVM. - Specialized algorithms like BalancedRandomForest.

Here's a mental model to keep in mind: the goal is to force the model to see the minority class as important and to evaluate success with metrics that don't reward the "always predict majority" cheat.

How it works step by step

Step 1: Detect class imbalance

Start by checking the distribution of your target variable:

import pandas as pd

df = pd.read_csv('creditcard.csv')
print(df['Class'].value_counts(normalize=True))

If the minority class is under ~20% of the data, you need to think about imbalance handling. Extreme imbalance (like 1% or less) demands stronger treatment.

Step 2: Choose your strategy

Based on your dataset size, noise level, and model type, pick from the approaches in the comparison table below. Often you'll combine data-level and algorithm-level tricks.

Step 3: Apply the chosen technique

For resampling, use imbalanced-learn (a scikit-learn add-on). For class weights, most scikit-learn models support class_weight='balanced'.

Step 4: Train and evaluate properly

Never rely on accuracy alone. Use confusion matrix, precision, recall, F1-score, and ROC-AUC.

Hands-on walkthrough

Let's walk through a complete example using the classic credit card fraud dataset. We'll compare a baseline model vs. one using class weights.

First, split your data (do this before any resampling to avoid data leakage):

from sklearn.model_selection import train_test_split

X = df.drop('Class', axis=1)
y = df['Class']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Baseline: no imbalance handling

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report

model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
print(classification_report(y_test, model.predict(X_test)))

Expected output (simplified):

              precision    recall  f1-score   support
           0       0.99      1.00      1.00     56861
           1       0.00      0.00      0.00       99

The model gets ~99% recall on the majority, but zero true positives. Useless for fraud detection.

Fix 1: class weights

from sklearn.model_selection import cross_val_score

weighted_model = LogisticRegression(max_iter=1000, class_weight='balanced')
weighted_model.fit(X_train, y_train)
print(classification_report(y_test, weighted_model.predict(X_test)))

# Check if it's better than random guessing
print("Cross-val ROC-AUC:", cross_val_score(weighted_model, X_train, y_train, scoring='roc_auc', cv=5).mean())

Fix 2: oversampling with SMOTE

Oversampling happens after the train/test split, on the training data only:

from imblearn.over_sampling import SMOTE

smote = SMOTE(random_state=42)
X_train_resampled, y_train_resampled = smote.fit_resample(X_train, y_train)

print("Class distribution after SMOTE:", y_train_resampled.value_counts())

smote_model = LogisticRegression(max_iter=1000)
smote_model.fit(X_train_resampled, y_train_resampled)
print(classification_report(y_test, smote_model.predict(X_test)))

Compare options / when to choose what

The table below compares the most common strategies for handling imbalanced datasets:

When to choose what:

• Extreme imbalance (<1%) → combine SMOTE with class-weight RF or BalancedRandomForest
• Almost800 training hyperparameter tuning for decision trees — you can set max_depth, and our earlier random forest did oversampling on the training split; the tree uses stratified splits by default. We should do the randomness properly: when next topic is uniform datasets, we can imagine a per-variable view.

Key principles to remember

• Many papers use Tomek links for undersampling (removing noisy edges), and it can be an alternative; but sticking to oversampling + class weights is the simplest robust pattern.
• Resampling should occur inside a pipeline so it doesn't leak test information; that’s covered in the next lesson about imbalanced cross-validation.