Combine Trees with Random Forests

Every data scientist has felt the sting: a single decision tree nails your training set but crumbles on new data, or worse, a tiny shift in the data produces an entirely different tree. You spend hours tuning depth and split criteria, and the model still feels like a fragile house of cards. That's the pain this lesson solves. You'll learn how combining trees with random forests turns a single, unstable predictor into a robust ensemble that reduces variance, fights overfitting, and delivers reliable accuracy with just a few lines of Python. By the end, you'll know the core idea, see it in action, and be ready to use it in your own projects.

The Problem This Lesson Solves

A single decision tree is a powerful non-linear model, but it's also famously high-variance. Change a few training samples and you might get a completely different tree structure, leading to dramatically different predictions. This instability causes two major headaches:

• Overfitting: the tree memorizes noise in the training data, so it performs poorly on unseen data.
• Inconsistent results: small perturbations in the data (or even the random seed) can change the model output substantially, making it hard to trust.

This is not just a theoretical concern. In real-world ML pipelines, a single tree's predictions can vary wildly across runs, making debugging and deployment frustrating. The problem gets worse as you add more features or use shallow trees that never capture complex patterns.

What if you could build many trees — each slightly different — and let them vote? That's exactly the insight behind random forests. Instead of relying on one fragile model, you aggregate the wisdom of a forest, smoothing out individual errors and producing a model that generalizes better.

Here's the core pain point: you need a model that is both expressive (captures non-linear relationships) and stable (doesn't overfit). A single tree fails on the second. Random forests solve this by combining many trees, each trained on a random subset of data and features, then averaging their predictions. The result: a model with lower variance, similar bias, and often better accuracy.

Core Concept / Mental Model

Think of random forests like a committee of experts. Each expert (tree) has a different background (random data sample) and focuses on different aspects of the problem (random feature subset). Individually, they might be biased or noisy, but together, their collective wisdom cancels out individual mistakes.

Analogy: Imagine you want to predict whether it will rain tomorrow. You ask 100 weather enthusiasts, each using slightly different data sources (some watch temperature, others pressure, others humidity). Each might be wrong sometimes, but if you take a majority vote, the result is usually reliable.

Key concepts to internalize:

• Ensemble learning: combining multiple models to improve performance.
• Bootstrap aggregating (bagging): each tree is trained on a random sample of the data (with replacement), called a bootstrap sample.
• Feature randomness: at each split, only a random subset of features is considered. This decorrelates the trees — if all trees used the same features, they'd make similar mistakes.
• Aggregation: for classification, you use majority voting; for regression, you average the predictions.

How it differs from a single tree:

The bias-variance tradeoff is central here. A single tree has low bias (it can fit any pattern) but high variance (it changes with the data). By averaging many trees, you keep bias roughly the same but slash variance. That's the magic of random forests — you get the expressiveness of a tree without the instability.

How It Works Step by Step

Let's break down the algorithm into concrete steps. This process is what sklearn's RandomForestClassifier or RandomForestRegressor does under the hood.

1. Bootstrap sampling (bagging): For each tree (say n_estimators=100), draw a random sample of n training examples with replacement. This means some samples appear multiple times, others not at all. Roughly 63% of the original data appears in each bootstrap sample.

2. Build a decision tree on each sample: Each tree is grown to its full depth (or until a minimum leaf size is reached). Crucially, at each split, instead of considering all features, only a random subset is examined. The size of this subset is typically sqrt(n_features) for classification and log2(n_features)+1 for regression.

3. Do not prune: Unlike a single tree, you typically do not prune a random forest's trees. The goal is to keep them deep and high-variance; the averaging will fix the variance.

4. Aggregate predictions: For classification, take a majority vote across all trees; for regression, take the mean of the predictions.

Why feature randomness matters: If every tree considered all features, the forest would be just a bag of similar trees — they'd all make the same mistakes, and averaging wouldn't help much. By randomly limiting the features at each split, you get decorrelated trees, which makes the ensemble's average more stable and accurate.

The math behind it (informally): For a set of independent, identically distributed random variables with variance σ², the variance of their average is σ²/n. Trees aren't fully independent, but feature randomness reduces correlation, so the variance reduction is significant.

Here's a simplified pseudocode flow:

For each tree in n_estimators:
    sample = bootstrap_sample(training_data)
    tree = build_decision_tree(sample, max_features=sqrt(n_features))
    add tree to forest

For a new sample x:
    predictions = [tree.predict(x) for tree in forest]
    # classification: return majority vote
    # regression: return mean(predictions)

Hands-On Walkthrough

Let's implement a random forest step by step. We'll use scikit-learn on a classic dataset — the Iris dataset for classification and the diabetes dataset for regression.

First, ensure you have scikit-learn installed:

pip install scikit-learn

Example 1: Classification with RandomForestClassifier

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

# Load data
iris = load_iris()
X, y = iris.data, iris.target

# Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create and train the forest
clf = RandomForestClassifier(n_estimators=100, random_state=42)
clf.fit(X_train, y_train)

# Predict and evaluate
preds = clf.predict(X_test)
print(f"Accuracy: {accuracy_score(y_test, preds):.3f}")

# Feature importance (average importance across trees)
print("Feature importances:", clf.feature_importances_)

Expected output (may vary slightly):

Accuracy: 1.000
Feature importances: [0.009 0.014 0.537 0.440]

The accuracy is 100% on the test set — a single tree might not achieve that consistently, but the forest does.

Example 2: Regression with RandomForestRegressor

from sklearn.datasets import load_diabetes
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestRegressor
import numpy as np

# Load data
diabetes = load_diabetes()
X, y = diabetes.data, diabetes.target

# Create model
reg = RandomForestRegressor(n_estimators=200, random_state=42)

# Cross-validate
scores = cross_val_score(reg, X, y, cv=5, scoring='neg_mean_squared_error')
rmse = np.sqrt(-scores)
print(f"RMSE per fold: {rmse}")
print(f"Mean RMSE: {rmse.mean():.3f} (+/- {rmse.std():.3f})")

Expected output:

RMSE per fold: [57.5 63.8 60.9 59.1 61.2]
Mean RMSE: 60.5 (+/- 2.1)

The low standard deviation across folds indicates stability.

Example 3: Tuning key hyperparameters

The two most important knobs are n_estimators and max_features. Let's see their effect.

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split

wine = load_wine()
X, y = wine.data, wine.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

for n in [1, 10, 50, 100, 200]:
    clf = RandomForestClassifier(n_estimators=n, random_state=0)
    clf.fit(X_train, y_train)
    acc = accuracy_score(y_test, clf.predict(X_test))
    print(f"n_estimators={n}: accuracy={acc:.3f}")

Expected output (approximately):

n_estimators=1: accuracy=0.870
n_estimators=10: accuracy=0.963
n_estimators=50: accuracy=0.963
n_estimators=100: accuracy=0.981
n_estimators=200: accuracy=0.981

Notice how accuracy improves and then plateaus — adding more trees beyond a certain point gives diminishing returns.

Compare Options / When to Choose What

Random forests aren't the only ensemble method. Here's how they compare with popular alternatives.

Choosing between random forest and gradient boosting:

• If you're in a hackathon or quick prototype, start with random forest — it works well out-of-the-box.
• If you're in production with strict latency requirements, a single tree might be faster, but random forest is still reasonable.
• If you have millions of rows and care about the last 0.5% accuracy, gradient boosting often wins, but requires more expertise.
• For feature importance insights, random forest gives reliable importance scores (though correlated features can distort them).

Variations of random forests:

• Extra Trees (ExtraTreesClassifier): randomness at the split threshold too, making trees even more diverse and sometimes faster.
• Isolation Forest (IsolationForest): a different use — anomaly detection, not prediction.
• Random Forest with class weights (class_weight='balanced'): handle imbalanced datasets.

Troubleshooting & Edge Cases

Even with random forests, things can go wrong. Here are common pitfalls and how to fix them.

Problem 1: Overfitting despite random forest

• Symptom: Training accuracy is near 100%, test accuracy is much lower.
• Cause: Trees are too deep and the forest is too large relative to your data. Also, if max_features is too large, trees are too correlated.
• Fix: Increase min_samples_leaf (e.g., 3–10), reduce max_depth (e.g., 3–5), and lower max_features. Use cross-validation to find good values.

Problem 2: The model takes forever to train

• Symptom: Training time is hours on a decent dataset.
• Cause: Too many trees, too deep, or too many features.
• Fix: Reduce n_estimators (100 is usually enough), set max_depth (e.g., 10), and use max_features='sqrt' for classification. Also, consider n_jobs=-1 to use all CPU cores.

Problem 3: Predictions are biased toward the majority class

• Symptom: On imbalanced data, the forest rarely predicts the minority class.
• Cause: The forest is optimizing overall accuracy, ignoring rare classes.
• Fix: Pass class_weight='balanced' (or 'balanced_subsample'), or use sample_weight during fitting. Also, you can adjust the decision threshold after training.

Problem 4: Feature importances are misleading

• Symptom: Features known to be irrelevant show high importance.
• Cause: Correlated features — the forest may distribute importance among them, or a feature that perfectly separates a small subset gets high importance by chance.
• Fix: Use permutation importance (sklearn.inspection.permutation_importance) which measures the drop in performance when a feature is shuffled. This is more reliable.

Problem 5: Different runs give different results

• Symptom: You get different accuracy each time you run your script.
• Cause: Randomness in bootstrap and feature selection.
• Fix: Set random_state=42 (or any fixed integer) for reproducibility, and consider using deterministic=1 if using newer scikit-learn versions.

What You Learned & What's Next

Let's recap what you've accomplished in this lesson:

• You understand that a single decision tree suffers from high variance and instability.
• You can explain the random forest mental model: a committee of expert trees, each trained on a bootstrap sample and a random feature subset, with predictions aggregated by voting or averaging.
• You know the step-by-step algorithm: bootstrap sampling, tree building with feature randomness, and aggregation.
• You've implemented classification and regression random forests using scikit-learn and tuned n_estimators.
• You can compare random forests with alternatives like gradient boosting and single trees, and choose the right approach.
• You've encountered common pitfalls like overfitting, slow training, class imbalance, misleading feature importance, and non-reproducibility — and you know how to fix them.

Your learning objectives achieved: - Explain the core idea behind combining trees with random forests. ✅ - Complete a practical exercise for combining trees with random forests. ✅

What's next? Now that you can build a forest, the natural next step is hyperparameter tuning and cross-validation. You'll learn how to systematically find the best n_estimators, max_depth, max_features, and min_samples_leaf using GridSearchCV and RandomizedSearchCV. This will take your random forest from "good enough" to "optimal." Stay tuned for the next lesson, where we'll dive into proper model selection techniques that you can apply to any ensemble model.

Until then, try this quick practice: take the Iris example and experiment with different max_features values (1, 2, 3, 4) and see how accuracy changes. Also, set random_state to different values and observe how stable the test accuracy is. This will solidify your intuition about the tradeoffs involved.