A/B Testing for Models

You've spent weeks training, fine-tuning, and evaluating a new model version — offline metrics look great, but the moment it hits production you see bizarre user behavior and no clear signal whether it's actually better than the one you shipped last quarter. Offline evaluations can't tell you how real users will react, and deploying a new model to 100% of traffic without evidence is a gamble your users will notice. That's why implement A/B testing for models is a required skill: it's the only way to compare model versions on live traffic with statistical confidence, catch regressions before they become disasters, and make data-driven decisions instead of trusting your gut.

The problem this lesson solves

In Applied AI engineering, model evaluation usually happens in a clean Jupyter notebook: you compute accuracy, F1, or ROUGE score on a holdout set, pat yourself on the back, and merge to production. But the real world is messy. Your offline dataset may be stale, your training distribution may differ from live traffic, or your model might optimize for the metric you chose while ignoring what users actually value — engagement, retention, or revenue. Without a way to test on real users, you're flying blind.

A/B testing solves this by comparing two or more model versions on live traffic while keeping everything else constant. You assign a subset of users to the new model (the treatment) and the rest to the current production model (the control), collect outcomes, and use statistics to answer one question: did the new model perform better, worse, or the same — and by how much? This lesson gives you the practical scaffolding to implement this in Python, from exposure logging to decision metrics to final analysis.

Why it matters now: As you progress through the Applied AI engineering track, you'll move from building models to maintaining them in production. A/B testing is the bridge between "works in my notebook" and "works in the real world."

Core concept / mental model

Think of A/B testing as a controlled experiment on your live system. Your model is the treatment, your users are the subjects, and the traffic split is your randomized assignment. The goal is to measure the causal effect of the new model on a business metric — not just correlation.

A simple mental model: imagine you run a restaurant and you want to test a new recipe. You don't change the recipe for everyone — you serve the new dish to a random half of customers and collect feedback. If the new recipe significantly improves satisfaction, you roll it out; if not, you keep the old one. Model A/B testing works exactly the same way, but with HTTP requests instead of plates of food.

Key terminology you need to master: - Control — the current production model (baseline) - Treatment — the new model candidate - Exposure — the moment a user is assigned to a variant - Metric — the quantifiable outcome you care about (e.g., click-through rate, conversion, user satisfaction) - Statistical significance — the probability that the observed difference is not due to random chance - Sample size — the number of exposures needed to detect a meaningful effect

Pro tip: You're not just comparing models; you're comparing decisions the models make. The same model with a different temperature setting is a new experiment.

How it works step by step

Implementing A/B testing for models in production involves a repeatable pipeline. Here's the logical sequence:

1. Define the hypothesis and metric. Decide what success looks like. For a recommendation model, it might be click-through rate; for a fraud detection model, it might be false positive rate at a fixed recall. Make sure your metric is measurable in real time.

2. Choose a traffic split. Allocate a percentage of traffic to each variant. A 50/50 split gives you the fastest statistical signal, but you may start with a smaller treatment group to limit risk (e.g., 10% new model, 90% control).

3. Log exposures. The moment a request is routed to a variant, log that event with a unique experiment ID, variant, user ID, and timestamp. This is your denominator for metric computation.

4. Compute the metric per variant. Aggregate outcomes (conversions, clicks, scores) per variant over a time window.

5. Run statistical analysis. Use a statistical test (e.g., two-proportion z-test for binary metrics, t-test for continuous metrics) to compute a p-value and confidence interval.

6. Decide and act. If the treatment wins with statistical significance and meets your practical threshold (e.g., ≥1% lift), roll it out. Otherwise, keep control and iterate.

Key principle: Randomization is critical. If you route traffic to variants based on user ID hash, a fixed portion of users consistently see one model — that's fine, but ensure the split is stable and not biased by demographics.

Hands-on walkthrough

Let's build a minimal but complete A/B testing system in Python. We'll simulate traffic, log exposures, and analyze the results.

Step 1: Simulate a simple A/B test for a binary metric

We'll simulate a scenario where the control model has a 5% conversion rate and the treatment model has a 7% conversion rate. We'll run 10,000 exposures per variant.

import numpy as np
from scipy.stats import norm

np.random.seed(42)

# True conversion probabilities
p_control = 0.05
p_treatment = 0.07

# Number of exposures per variant
n_per_variant = 10000

# Simulate individual outcomes (1 = conversion, 0 = no)
control_outcomes = np.random.binomial(1, p_control, n_per_variant)
treatment_outcomes = np.random.binomial(1, p_treatment, n_per_variant)

# Observed conversion rates
control_rate = control_outcomes.mean()
treatment_rate = treatment_outcomes.mean()
print(f"Control conversion rate: {control_rate:.4f}")
print(f"Treatment conversion rate: {treatment_rate:.4f}")

Expected output (varies slightly):

Control conversion rate: 0.0511
Treatment conversion rate: 0.0716

Step 2: Perform a two-proportion z-test

We'll test whether the difference is statistically significant.

# Number of successes and trials
control_success = control_outcomes.sum()
treatment_success = treatment_outcomes.sum()
control_n = n_per_variant
treatment_n = n_per_variant

# Pooled conversion rate
pooled_p = (control_success + treatment_success) / (control_n + treatment_n)

# Standard error
se = np.sqrt(pooled_p * (1 - pooled_p) * (1 / control_n + 1 / treatment_n))

# Z statistic
z_stat = (treatment_rate - control_rate) / se

# Two-tailed p-value
p_value = 2 * (1 - norm.cdf(abs(z_stat)))

# Confidence interval (95%)
z_critical = norm.ppf(0.975)
ci_lower = (treatment_rate - control_rate) - z_critical * se
ci_upper = (treatment_rate - control_rate) + z_critical * se

print(f"Z-statistic: {z_stat:.3f}")
print(f"P-value: {p_value:.4f}")
print(f"95% CI for lift: [{ci_lower:.4f}, {ci_upper:.4f}]")
print("Statistically significant?", p_value < 0.05)

Expected output:

Z-statistic: 6.604
P-value: 0.0000
95% CI for lift: [0.0142, 0.0268]
Statistically significant? True

Step 3: Logging exposures in a real system

In production, you'd log each exposure to a structured log. Here's a minimal Python snippet for a Flask-like endpoint.

import hashlib
import json
import time
import random

def route_to_variant(user_id: str, experiment_name: str, variants: list, split: list):
    """Route a user to a variant deterministically or randomly."""
    # Use hashed user_id for stable allocation (optional)
    if split is None:
        return random.choice(variants)
    hash_hex = hashlib.sha256(f"{user_id}:{experiment_name}".encode()).hexdigest()
    bucket = int(hash_hex, 16) % 100
    cumulative = 0
    for variant, perc in zip(variants, split):
        cumulative += perc
        if bucket < cumulative:
            return variant
    return variants[-1]

# Example usage
experiment = "recommendation-v2-vs-v1"
variant = route_to_variant("user-123", experiment, ["control", "treatment"], [50, 50])

# Log exposure (pseudo-code)
log_entry = {
    "event": "exposure",
    "experiment": experiment,
    "variant": variant,
    "user_id": "user-123",
    "timestamp": time.time()
}
print(json.dumps(log_entry, indent=2))

Output:

{
  "event": "exposure",
  "experiment": "recommendation-v2-vs-v1",
  "variant": "treatment",
  "user_id": "user-123",
  "timestamp": 1712366543.123456
}

Step 4: Continuous metric analysis

For metrics like average session duration, use a t-test.

from scipy.stats import ttest_ind

# Simulate continuous metric (e.g., time spent in seconds)
control_times = np.random.normal(100, 15, n_per_variant)
treatment_times = np.random.normal(105, 15, n_per_variant)

t_stat, p_val = ttest_ind(treatment_times, control_times, equal_var=False)
print(f"T-statistic: {t_stat:.3f}, P-value: {p_val:.4f}")

Output:

T-statistic: 23.196, P-value: 0.0000

Compare options / when to choose what

There are several approaches to online evaluation beyond classic A/B tests. Here's a comparison to help you choose:

Pro tip: Start with a classic A/B test for simplicity and interpretability. If you need faster results, consider interleaving for recommendation systems or a bandit for continuous optimization.

Troubleshooting & edge cases

Edge cases will bite you. Here are common issues and fixes:

• P-value looks significant but the lift is tiny. Statistical significance ≠ practical significance. Define a minimum effect size before the experiment.
• Traffic split is 50/50, but treatment group has 10% more users. Check your hashing function — make sure the allocation is stable and uniform.
• Metric is highly skewed. For revenue or session length, the t-test assumes normality. Use a Mann-Whitney U test or log-transform the metric.
• Users switch devices. A user may see the control on mobile and treatment on desktop. Use a unique user ID and log it consistently, or restrict to one device per experiment.
• Multiple simultaneous experiments. Users in both experiments can interact, biasing results. Use exclusive cohorts or account for interactions in analysis.
• Detecting a real effect takes too long. Use a power analysis beforehand to estimate required sample size. Increase traffic split or choose a more sensitive metric.

Here's a quick power analysis snippet:

from statsmodels.stats.power import NormalIndPower

power_analysis = NormalIndPower()
# Detect a 20% relative lift (0.05 -> 0.06) with 80% power, alpha 5%
sample_size_per_group = power_analysis.solve_power(effect_size=0.01/0.02, alpha=0.05, power=0.8, alternative='two-sided')
print(f"Required per group: {int(sample_size_per_group)}")

Typically this outputs several thousand, so plan accordingly.

What you learned & what's next

You now understand the core concept of A/B testing for models: comparing two versions on live traffic with statistical rigor. You've implemented a basic pipeline — exposure logging, metric computation, and hypothesis testing using a z-test and t-test — and you know how to troubleshoot common issues like small effects, skewed metrics, and device switching.

You've met both learning objectives: you can explain why A/B testing is essential and apply it via Python simulation. Next in the Applied AI engineering track, you'll explore multi-armed bandits as an adaptive alternative to A/B testing, allowing you to automatically allocate more traffic to the better-performing model over time — a natural evolution of the experiment framework you just built.

Keep experimenting!