Explain Predictions with SHAP

You've trained a model that hits impressive accuracy, but when a stakeholder asks why it made a specific decision, you freeze. Accuracy alone doesn't build trust — it doesn't tell you which features drove a loan denial, a fraud flag, or a medical diagnosis. Without a way to explain predictions, your model is a black box: technically powerful, but practically risky. That's the exact pain point this lesson solves: you'll learn to explain predictions with SHAP, the industry-standard framework for turning any model's outputs into clear, human-readable explanations.

The problem this lesson solves

Machine learning models are notorious for being opaque. A gradient-boosted tree or a deep neural network makes decisions through thousands of invisible interactions. When a customer asks why their application was rejected, or a compliance auditor demands a rationale, a simple accuracy score answers nothing.

This is where model interpretability becomes a hard requirement, not a nice-to-have. Regulations like GDPR in Europe and algorithms accountability laws in North America increasingly require that automated decisions be explainable. Even outside legal mandates, debugging models requires knowing which features drove the errors.

SHAP — SHapley Additive exPlanations — solves this by assigning each feature a contribution score for each prediction. It answers the question: "How much did each input feature push the prediction away from the baseline?" By the end of this lesson, you'll be able to:

• Compute SHAP values for any trained model
• Visualize global and local explanations
• Use those insights to debug, validate, and communicate model behavior confidently

Core concept / mental model

Think of SHAP values like a salary negotiation with a committee. A baseline prediction (say, the average predicted salary) exists before any feature is considered. Each committee member — representing a feature like age, income, credit_score — argues how much they should adjust that baseline. The SHAP value is the final agreed-upon contribution from each member, such that all contributions sum exactly to the difference between the prediction and the baseline.

More formally, SHAP values come from cooperative game theory, specifically the Shapley value concept. For each prediction, SHAP computes a fair allocation of the prediction among the features, considering every possible combination of features. This is computationally heavy, but SHAP provides optimized implementations for common model types (tree-based, linear, deep) that make it feasible in practice.

Key properties that make SHAP trustworthy:

• Additivity: The sum of SHAP values plus the baseline equals the model's prediction.
• Consistency: If a model changes so a feature matters more, that feature's SHAP value never decreases.
• Local accuracy: The explanation is accurate for the specific prediction.

When you visualize SHAP values, you get two complementary perspectives: local explanations (why a single prediction was made) and global explanations (which features matter most across the whole dataset).

How it works step by step

The process of explaining predictions with SHAP follows a clear pipeline:

1. Train your model — SHAP works on any trained model, regardless of algorithm.
2. Choose an explainer — SHAP provides specialized explainers: TreeExplainer for tree-based models, LinearExplainer for linear models, KernelExplainer for any model, and DeepExplainer for neural networks.
3. Compute SHAP values — The explainer calculates contribution scores for each feature per sample.
4. Interpret locally — Examine a single prediction's SHAP values to see which features pushed the outcome up or down.
5. Interpret globally — Aggregate SHAP values across the dataset to rank feature importance and understand typical behavior.
6. Visualize — Use SHAP's built-in plots (waterfall, beeswarm, bar) to communicate findings.

The crucial cause-and-effect relationship: higher absolute SHAP value = stronger influence on the prediction. Positive values push the prediction up; negative values push it down.

Hands-on walkthrough

Let's apply this to a real dataset. We'll train a gradient boosting model on the classic California housing dataset and explain its predictions with SHAP. First, install the required packages:

pip install shap scikit-learn xgboost

Step 1: Train a model

import shap
import xgboost as xgb
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split

# Load data
data = fetch_california_housing()
X = data.data
y = data.target
feature_names = data.feature_names

# Split
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# Train a gradient boosting model
model = xgb.XGBRegressor(n_estimators=100, max_depth=4, random_state=42)
model.fit(X_train, y_train)

print("Model trained. Validation R²:", round(model.score(X_val, y_val), 3))

Expected output:

Model trained. Validation R²: 0.833

Step 2: Create a SHAP explainer and compute values

# Use TreeExplainer (fast and exact for tree models)
explainer = shap.TreeExplainer(model)

# Compute SHAP values for the validation set (first 100 rows to keep it fast)
shap_values = explainer.shap_values(X_val[:100])

print("SHAP values shape:", shap_values.shape)
print("Baseline (expected value):", explainer.expected_value)

Expected output:

SHAP values shape: (100, 9)
Baseline (expected value): 2.067

Step 3: Visualize a single prediction's explanation

# Explain the first prediction
shap.waterfall_plot(shap.Explanation(values=shap_values[0],
                                      base_values=explainer.expected_value,
                                      data=X_val[0],
                                      feature_names=feature_names))

The waterfall plot shows the baseline value plus each feature's contribution, stacking up to the final prediction. For example, if MedInc (median income) has a SHAP value of +0.5, it pushed the predicted house price up by $50,000.

Step 4: View global feature importance

# Summary bar plot — mean absolute SHAP values across all samples
shap.summary_plot(shap_values, X_val[:100], feature_names=feature_names, plot_type="bar")

Expected output: a bar chart showing MedInc at the top as the most influential feature, followed by Latitude and Longitude.

Step 5: Use SHAP in production

import joblib

# Save the model and explainer for production use
joblib.dump(model, "model.pkl")
joblib.dump(explainer, "explainer.pkl")

# Later, load and explain a new prediction
loaded_model = joblib.load("model.pkl")
loaded_explainer = joblib.load("explainer.pkl")
new_sample = X_val[42].reshape(1, -1)

shap_values_new = loaded_explainer.shap_values(new_sample)
print("SHAP values for new sample:", shap_values_new)

Compare options / when to choose what

SHAP is not the only interpretability tool. Here's how it stacks up against alternatives:

Pro tip: For tree-based models, always prefer TreeExplainer — it's exact and dramatically faster than KernelExplainer. For neural networks, opt for DeepExplainer or GradientExplainer. If your model is a custom black box, KernelExplainer works but is slower.

When to choose SHAP over others:

• When you need both local and global explanations
• When regulatory compliance demands a consistent, theoretically sound method
• When you need to debug model behavior across feature interactions
• When your stakeholder wants to understand why a decision was made, not just which features matter on average

Troubleshooting & edge cases

1. SHAP values take too long to compute

Cause: KernelExplainer on a large dataset, or tree explainer on a very large model.\ Fix: Use TreeExplainer for trees; sample your data (e.g., 1000 rows) for explanation; disable profiling or use GPU acceleration if available.

2. Duplicate features cause "looks like the feature was split" error

Cause: Your dataset has correlated or duplicate columns.\ Fix: Drop duplicates before training and explaining. Use shap.utils.check_additivity() to verify.

3. SHAP values don't sum to prediction

Cause: Usually because the baseline (expected_value) is the mean prediction over the training set. The sum of SHAP values plus that mean equals the prediction, but if you use a different baseline, the equation breaks.\ Fix: Use the explainer.expected_value provided by the explainer, not a manually computed mean.

4. TreeExplainer fails on non-tree models

Cause: SHAP's tree explainer only works with tree-based models (sklearn trees, xgboost, lightgbm, catboost).\ Fix: Use KernelExplainer or PermutationExplainer (a faster approximation) for arbitrary models.

5. Interpreting SHAP values with one-hot encoded categorical features

Cause: One-hot features are treated independently, which can obscure the original category's effect.\ Fix: Encode categories as integers or use a custom feature_names mapping, and rely on shap.summary_plot which groups one-hot columns if you pass a pandas DataFrame with categorical dtypes.

What you learned & what's next

You now have a complete mental model and practical toolkit for explaining predictions with SHAP. Specifically, you can:\n - Explain why a single prediction was made using waterfall plots and SHAP values\n- Analyze global feature importance to audit model behavior\n- Choose the right explainer based on model type and dataset size\n- Troubleshoot common SHAP pitfalls in real projects\n The ability to interpret models is a cornerstone of responsible AI development. With SHAP in your arsenal, you can move from "black box" deployment to transparent, defensible AI systems. This skill directly prepares you for the next lesson in the Applied AI engineering track: evaluating and monitoring model explanations — where you'll learn how to use SHAP values to detect data drift and build automated explanation checks into your ML pipelines.