A/B testing & experimentation
User bucketing, experiment metrics, statistical comparison, and rollout guardrails.
Bayesian A/B Test Credible Interval in Python
Simulates A/B test data and computes posterior credible intervals and the probability that variant B outperforms A using Bayesian Beta-Binomial inference.
import numpy as np
from scipy import stats
# Simulated A/B test data
n_A = 1000
n_B = 1000
conversions_A = 120
conversions_B = 140
# Prior: Beta(1, 1) uniform
alpha_prior, beta_prior = 1, 1
# Posterior parameters
alpha_A = alpha_prior + conversions_A
beta_A = beta_prior + n_A - conversions_A
alpha_B = alpha_prior +…Check Covariate Balance in Python
Compute standardized mean differences and KS tests to check covariate balance between treatment and control groups in Python.
import numpy as np
from scipy import stats
def balance_check(treatment, covariate):
"""Check covariate balance between treatment and control groups."""
treat_vals = covariate[treatment == 1]
control_vals = covariate[treatment == 0]
# Standardized mean difference
pooled_std = np.sqrt((np.var(t…Check Sample Ratio Mismatch in Python
Estimates the probability that a simple random sample's proportion differs from the population proportion by more than 10% using simulation.
import random
def sample_ratio_mismatch(population_size: int, sample_size: int, p: float) -> float:
"""
Estimate the probability that a simple random sample's proportion
differs from the population proportion by more than 10%.
"""
total_counts = [0, 0]
for _ in range(10000):
sample = …Chi-Square Test in Python for Conversion Mock Data
Compute the chi-square statistic and approximate p-value for a mock A/B conversion test using the standard library.
import math
from collections import Counter
def chi_square_statistic(observed):
"""
Compute chi-square statistic for a mock conversion test.
observed: dict mapping outcomes to observed frequencies.
"""
observed = Counter(observed)
n = sum(observed.values())
expected = n / len(observed) if …Delta Method for Ratio Metrics in A/B Testing with Python
Computes the confidence interval for the difference between two ratio metrics using the delta method, with mock A/B test data.
import numpy as np
from scipy.stats import norm
def delta_method_ratio_delta(control: np.ndarray, treatment: np.ndarray, confidence: float = 0.95):
"""Estimate confidence interval for ratio metric using delta method.
Args:
control: numerator/denominator pairs from control group (n x 2 array)
…Difference in Differences Mock in Python
Generate mock panel data with a known treatment effect and compute a difference-in-differences estimate using group and period means.
import numpy as np
import pandas as pd
# Generate mock panel data: 2 groups (control=0, treatment=1) × 2 periods (pre=0, post=1)
rng = np.random.default_rng(42)
n_per_cell = 50
data = []
for group in [0, 1]:
for period in [0, 1]:
# True effect: treatment increases outcome by 5 in the post period
…Epsilon Greedy Bandit Mock in Python
A simple epsilon-greedy multi-armed bandit simulation that balances exploration and exploitation to estimate true means of several Bernoulli-like reward distributions.
import random
class Bandit:
def __init__(self, true_mean):
self.true_mean = true_mean
self.estimated_mean = 0.0
self.n_pulls = 0
def pull(self):
return random.gauss(self.true_mean, 1.0)
def update(self, reward):
self.n_pulls += 1
self.estimated_mean += (r…How to Build a Guardrail Metrics Monitor in Python
This code implements a mock monitor that records metric values, checks them against thresholds, and summarizes pass/alert statistics.
import random
import time
from collections import defaultdict
class GuardrailMetricsMonitor:
def __init__(self):
self.metrics = defaultdict(list)
self.thresholds = {
"prompt_toxicity": 0.8,
"response_length": 500,
"latency_ms": 1000,
}
def record(s…How to Calculate Minimum Sample Size for a T-Test in Python
Compute the minimum sample size per group for a two-sample t-test using effect size, significance level, and statistical power.
import math
from scipy.stats import norm
def min_sample_size(effect_size, alpha=0.05, power=0.8):
"""
Calculate minimum sample size for a two-sample t-test (equal groups).
Args:
effect_size: Cohen's d (standardized mean difference)
alpha: significance level (Type I error)
power: …How to Compute CUPED Variance Reduction in Python
Implement CUPED in Python to reduce variance of A/B test treatment effect estimates using pre-experiment covariates.
import numpy as np
def compute_cuped_reduction(control, variant, covariate):
"""
Compute variance reduction using CUPED (Controlled Experiment with
Pre-Experiment Data). Uses pre-experiment covariate values to
reduce variance of the treatment effect estimate.
"""
control = np.asarray(control, …How to Compute Mann-Whitney U Test in Python
Compute the Mann-Whitney U statistic and p-value manually in Python with tie correction and a normal approximation for independent samples.
import numpy as np
from scipy import stats
def mann_whitney_u_mock(sample_a, sample_b):
"""Compute Mann-Whitney U and p-value manually."""
# Combine and rank
combined = sample_a + sample_b
n_a, n_b = len(sample_a), len(sample_b)
n_total = n_a + n_b
# Rank with ties handling (average ranks…How to Create a Mock That Returns Inverse Counter Values in Python
Builds a Mock whose side_effect returns the inverse (1/count) of each Counter value, defaulting to 0.0 for unseen keys.
from collections import Counter
from unittest.mock import Mock
def inverse_mock(counter: Counter) -> Mock:
"""
Return a Mock that mimics the inverse of a Counter:
each key returns a value representing the inverse of its count.
The Mock's side_effect maps keys to their inverse counts.
"""
mock …How to Create an Interrupted Time Series Mock in Python
Generate simulated interrupted time series data with a pre/post-intervention trend, level shift, and noise to test segmented regression models.
import numpy as np
# Mock interrupted time series data
np.random.seed(42)
n_pre = 50
n_post = 50
time = np.arange(0, n_pre + n_post)
# Pre-intervention: linear trend + noise
pre_trend = 0.05 * time[:n_pre] + np.random.normal(0, 0.5, n_pre)
# Post-intervention: new slope + level shift + noise
post_trend = 0.05 * tim…How to Define a Mock Primary Metric in Python
Define a mock primary metric object with a name, value, and unit, and serialize it to a dictionary for experimentation and testing.
class Metric:
def __init__(self, name, value, unit=None):
self.name = name
self.value = value
self.unit = unit
def to_dict(self):
result = {"name": self.name, "value": self.value}
if self.unit:
result["unit"] = self.unit
return result
def __repr…How to Do Random Assignment in Python for A/B Tests
Assign each item to a binary group (0 or 1) with uniform probability using a small reusable function, optionally weighted, for A/B testing mocks.
import random
def random_assignment_uniform_mock(items, weights=None):
"""Assign each item to a group (0 or 1) with uniform probability."""
if weights is None:
# Default: each item independently gets 0 or 1 with 50% probability
return [random.randint(0, 1) for _ in items]
# Optional weight…How to Generate an Orthogonal Array for A/B Testing in Python
Generate a mock orthogonal array for multi-layer experiments with NumPy, ensuring balanced level combinations across experiment groups.
import numpy as np
def orthogonal_mock_layers(n_experiments: int, n_layers: int, n_levels: int) -> np.ndarray:
"""Generate an orthogonal array for multi-layer experiment design using base-level logic."""
ortho = np.indices((n_levels,) * n_layers).reshape(n_layers, -1).T
ortho = ortho % n_levels # Classic…How to Hash a User ID to an Experiment Bucket in Python
Deterministically map a user ID to one of N experiment buckets using MD5 hashing and modulo arithmetic.
import hashlib
def hash_to_bucket(user_id: str, num_buckets: int = 10) -> int:
"""Deterministically map a user_id to a bucket (0 to num_buckets-1)."""
digest = hashlib.md5(user_id.encode("utf-8")).hexdigest()
return int(digest[:8], 16) % num_buckets
if __name__ == "__main__":
# Mock experiment: split…How to Mock Mutual Exclusion for A/B Experiment Groups in Python
Simulate mutual exclusion for experiment groups using a thread-safe lock, ensuring only one member updates the shared counter at a time.
import threading
import time
import random
class CountingGate:
"""A mock mutual exclusion gate using a lock."""
def __init__(self):
self.counter = 0
self.lock = threading.Lock()
def enter(self, group_id, member_id):
with self.lock:
current = self.counter
t…How to Mock Stratified Assignment by Segment in Python
Simulate stratified assignment for A/B experiments by sampling a fixed proportion of units from each segment, with deterministic seeds for reproducibility.
import random
def stratified_assignment(segments, seed=None):
"""
Mock stratified assignment: given a dict of segment -> population size,
return a dict of segment -> sampled unit ids (deterministic with seed).
"""
if seed is not None:
random.seed(seed)
rng = random.Random(seed)
res…How to Mock a Confidence Interval for a Proportion in Python
Simulate a Bernoulli sample and compute a 95% confidence interval for a proportion using the normal approximation in Python.
import random
import math
def mock_ci(n=100, p_true=0.5, z=1.96, seed=42):
"""Simulate a sample proportion and compute its 95% confidence interval."""
random.seed(seed)
successes = sum(1 for _ in range(n) if random.random() < p_true)
p_hat = successes / n
se = math.sqrt(p_hat * (1 - p_hat) / n)
…How to Perform Intent-to-Treat Analysis in Python
Runs an intent-to-treat analysis on mock A/B test data, comparing outcomes by initial group assignment with a t-test for significance.
import pandas as pd
import numpy as np
def intent_to_treat_analysis(data):
"""Perform intent-to-treat (ITT) analysis.
ITT compares outcomes based on initial treatment assignment,
regardless of whether participants actually received the treatment.
"""
# Create a copy to avoid mutating the origina…How to Perform Welch's t-Test in Python
Calculate the Welch t-statistic and degrees of freedom for two samples with unequal variances using Python's statistics module.
import math
from statistics import mean, variance
def welch_t_test(sample1, sample2):
n1, n2 = len(sample1), len(sample2)
mean1, mean2 = mean(sample1), mean(sample2)
var1, var2 = variance(sample1), variance(sample2)
# Welch's t statistic
t_stat = (mean1 - mean2) / math.sqrt(var1 / n1 + var2 / n2…How to Run a Fisher Exact Test in Python
Compute the two-sided Fisher exact test p-value for a 2x2 contingency table using pure Python and the math module.
from math import comb, factorial
from itertools import combinations
def hypergeometric_probability(a, b, c, d):
"""Probability of observing table [[a, b], [c, d]] under the null."""
row1 = a + b
row2 = c + d
col1 = a + c
col2 = b + d
total = row1 + row2
return (comb(row1, a) * comb(row2, …How to Run a Permutation Test in Python
Run a Monte Carlo permutation test to compute a p-value for comparing two group means without parametric assumptions.
import random
import statistics
def permutation_test(group_a, group_b, n_permutations=10000, seed=42):
random.seed(seed)
combined = group_a + group_b
observed_diff = abs(statistics.mean(group_a) - statistics.mean(group_b))
count = 0
n = len(group_a)
for _ in range(n_permutations):
…Browse by section
Each section groups closely related Python snippets.
A/B testing & experimentation — Python code examples
What you will find here
This page collects a/b testing & experimentation snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.