A/B testing & experimentation
User bucketing, experiment metrics, statistical comparison, and rollout guardrails.
Bayesian A/B Test Credible Interval in Python
Simulates A/B test data and computes posterior credible intervals and the probability that variant B outperforms A using Bayesian Beta-Binomial inference.
import numpy as np
from scipy import stats
# Simulated A/B test data
n_A = 1000
n_B = 1000
conversions_A = 120
conversions_B = 140
# Prior: Beta(1, 1) uniform
alpha_prior, beta_prior = 1, 1
# Posterior parameters
alpha_A = alpha_prior + conversions_A
beta_A = beta_prior + n_A - conversions_A
alpha_B = alpha_prior +…Benjamini Hochberg FDR Correction in Python
Implement the Benjamini-HHochberg false discovery rate (FDR) procedure in Python to control the expected proportion of false positives among rejected hypotheses.
import numpy as np
def benjamini_hochberg(p_values, alpha=0.05):
p_values = np.array(p_values)
n = len(p_values)
sorted_idx = np.argsort(p_values)
sorted_p = p_values[sorted_idx]
thresholds = (np.arange(1, n + 1) / n) * alpha
significant = sorted_p <= thresholds
if not significan…Check Covariate Balance in Python
Compute standardized mean differences and KS tests to check covariate balance between treatment and control groups in Python.
import numpy as np
from scipy import stats
def balance_check(treatment, covariate):
"""Check covariate balance between treatment and control groups."""
treat_vals = covariate[treatment == 1]
control_vals = covariate[treatment == 0]
# Standardized mean difference
pooled_std = np.sqrt((np.var(t…Check Sample Ratio Mismatch in Python
Estimates the probability that a simple random sample's proportion differs from the population proportion by more than 10% using simulation.
import random
def sample_ratio_mismatch(population_size: int, sample_size: int, p: float) -> float:
"""
Estimate the probability that a simple random sample's proportion
differs from the population proportion by more than 10%.
"""
total_counts = [0, 0]
for _ in range(10000):
sample = …Chi-Square Test in Python for Conversion Mock Data
Compute the chi-square statistic and approximate p-value for a mock A/B conversion test using the standard library.
import math
from collections import Counter
def chi_square_statistic(observed):
"""
Compute chi-square statistic for a mock conversion test.
observed: dict mapping outcomes to observed frequencies.
"""
observed = Counter(observed)
n = sum(observed.values())
expected = n / len(observed) if …Delta Method for Ratio Metrics in A/B Testing with Python
Computes the confidence interval for the difference between two ratio metrics using the delta method, with mock A/B test data.
import numpy as np
from scipy.stats import norm
def delta_method_ratio_delta(control: np.ndarray, treatment: np.ndarray, confidence: float = 0.95):
"""Estimate confidence interval for ratio metric using delta method.
Args:
control: numerator/denominator pairs from control group (n x 2 array)
…Difference in Differences Mock in Python
Generate mock panel data with a known treatment effect and compute a difference-in-differences estimate using group and period means.
import numpy as np
import pandas as pd
# Generate mock panel data: 2 groups (control=0, treatment=1) × 2 periods (pre=0, post=1)
rng = np.random.default_rng(42)
n_per_cell = 50
data = []
for group in [0, 1]:
for period in [0, 1]:
# True effect: treatment increases outcome by 5 in the post period
…Epsilon Greedy Bandit Mock in Python
A simple epsilon-greedy multi-armed bandit simulation that balances exploration and exploitation to estimate true means of several Bernoulli-like reward distributions.
import random
class Bandit:
def __init__(self, true_mean):
self.true_mean = true_mean
self.estimated_mean = 0.0
self.n_pulls = 0
def pull(self):
return random.gauss(self.true_mean, 1.0)
def update(self, reward):
self.n_pulls += 1
self.estimated_mean += (r…Generate a Mock Multi-Armed Bandit Report in Python
Simulate a multi-armed bandit experiment with random pulls and rewards, then output a JSON report with per-arm statistics.
import random
import json
def generate_mock_bandit_report(num_arms=5, num_rounds=100, seed=42):
random.seed(seed)
arms = ["A", "B", "C", "D", "E"][:num_arms]
true_means = {arm: random.uniform(0.3, 0.7) for arm in arms}
pulls = {arm: 0 for arm in arms}
rewards = {arm: 0 for arm in arms}
for _ …How to Build a Guardrail Metrics Monitor in Python
This code implements a mock monitor that records metric values, checks them against thresholds, and summarizes pass/alert statistics.
import random
import time
from collections import defaultdict
class GuardrailMetricsMonitor:
def __init__(self):
self.metrics = defaultdict(list)
self.thresholds = {
"prompt_toxicity": 0.8,
"response_length": 500,
"latency_ms": 1000,
}
def record(s…How to Build a Simple Binary Protocol Parser Mock in Python
Defines a mock binary protocol with field definitions, encoding, and decoding to simulate network packet parsing for A/B testing and experiment setup.
class SimpleProtocol:
def __init__(self, name, version):
self.name = name
self.version = version
self.fields = []
def add_field(self, field_name, field_size):
self.fields.append((field_name, field_size))
def parse(self, data):
if len(data) != sum(size for _, size i…How to Calculate Minimum Sample Size for a T-Test in Python
Compute the minimum sample size per group for a two-sample t-test using effect size, significance level, and statistical power.
import math
from scipy.stats import norm
def min_sample_size(effect_size, alpha=0.05, power=0.8):
"""
Calculate minimum sample size for a two-sample t-test (equal groups).
Args:
effect_size: Cohen's d (standardized mean difference)
alpha: significance level (Type I error)
power: …How to Compute CUPED Variance Reduction in Python
Implement CUPED in Python to reduce variance of A/B test treatment effect estimates using pre-experiment covariates.
import numpy as np
def compute_cuped_reduction(control, variant, covariate):
"""
Compute variance reduction using CUPED (Controlled Experiment with
Pre-Experiment Data). Uses pre-experiment covariate values to
reduce variance of the treatment effect estimate.
"""
control = np.asarray(control, …How to Compute Mann-Whitney U Test in Python
Compute the Mann-Whitney U statistic and p-value manually in Python with tie correction and a normal approximation for independent samples.
import numpy as np
from scipy import stats
def mann_whitney_u_mock(sample_a, sample_b):
"""Compute Mann-Whitney U and p-value manually."""
# Combine and rank
combined = sample_a + sample_b
n_a, n_b = len(sample_a), len(sample_b)
n_total = n_a + n_b
# Rank with ties handling (average ranks…How to Conduct a Two-Sample T-Test in Python
Performs Welch's t-test for two independent samples, computing the t-statistic, degrees of freedom, and p-value using NumPy and SciPy.
import numpy as np
def two_sample_t_test(sample1, sample2):
"""Perform Welch's t-test for two independent samples."""
n1, n2 = len(sample1), len(sample2)
mean1, mean2 = np.mean(sample1), np.mean(sample2)
var1, var2 = np.var(sample1, ddof=1), np.var(sample2, ddof=1)
# Standard error of difference
…How to Create a Mock That Returns Inverse Counter Values in Python
Builds a Mock whose side_effect returns the inverse (1/count) of each Counter value, defaulting to 0.0 for unseen keys.
from collections import Counter
from unittest.mock import Mock
def inverse_mock(counter: Counter) -> Mock:
"""
Return a Mock that mimics the inverse of a Counter:
each key returns a value representing the inverse of its count.
The Mock's side_effect maps keys to their inverse counts.
"""
mock …How to Create a Sticky Consistent Mock with unittest.mock in Python
Shows how to use unittest.mock.patch.object to mock a method consistently across multiple calls, returning a sticky value every time.
from unittest.mock import patch
class Database:
def fetch(self, key):
return f"real value for {key}"
def get_value(db, key):
return db.fetch(key)
if __name__ == "__main__":
db = Database()
with patch.object(db, "fetch", return_value="sticky value") as mock_fetch:
result1 = get_value(…How to Create an Interrupted Time Series Mock in Python
Generate simulated interrupted time series data with a pre/post-intervention trend, level shift, and noise to test segmented regression models.
import numpy as np
# Mock interrupted time series data
np.random.seed(42)
n_pre = 50
n_post = 50
time = np.arange(0, n_pre + n_post)
# Pre-intervention: linear trend + noise
pre_trend = 0.05 * time[:n_pre] + np.random.normal(0, 0.5, n_pre)
# Post-intervention: new slope + level shift + noise
post_trend = 0.05 * tim…How to Define a Mock Primary Metric in Python
Define a mock primary metric object with a name, value, and unit, and serialize it to a dictionary for experimentation and testing.
class Metric:
def __init__(self, name, value, unit=None):
self.name = name
self.value = value
self.unit = unit
def to_dict(self):
result = {"name": self.name, "value": self.value}
if self.unit:
result["unit"] = self.unit
return result
def __repr…How to Do Random Assignment in Python for A/B Tests
Assign each item to a binary group (0 or 1) with uniform probability using a small reusable function, optionally weighted, for A/B testing mocks.
import random
def random_assignment_uniform_mock(items, weights=None):
"""Assign each item to a group (0 or 1) with uniform probability."""
if weights is None:
# Default: each item independently gets 0 or 1 with 50% probability
return [random.randint(0, 1) for _ in items]
# Optional weight…How to Evaluate Feature Flags in Python
A Python function that evaluates boolean feature flags with user-specific overrides, returning whether a flag is enabled and the reason for the decision.
import json
def evaluate_feature_flag(feature_name, context, flag_configs):
"""
Evaluates a boolean feature flag given a context dictionary.
Args:
feature_name: The name of the feature flag.
context: A dictionary of user/request context (e.g., {"user_id": "123"}).
flag_configs: A …How to Generate Multivariate JSON Mock Data in Python
This script generates mock multivariate JSON-compatible data with measurements and boolean flags for testing and experimentation pipelines.
import json
def multivariate_mock(row_count: int = 3) -> list:
"""Generate mock multivariate data as list of JSON-compatible dicts."""
records = []
for i in range(row_count):
record = {
"id": i + 1,
"measurements": {
"temperature": 20.5 + i * 1.5,
…How to Generate an Orthogonal Array for A/B Testing in Python
Generate a mock orthogonal array for multi-layer experiments with NumPy, ensuring balanced level combinations across experiment groups.
import numpy as np
def orthogonal_mock_layers(n_experiments: int, n_layers: int, n_levels: int) -> np.ndarray:
"""Generate an orthogonal array for multi-layer experiment design using base-level logic."""
ortho = np.indices((n_levels,) * n_layers).reshape(n_layers, -1).T
ortho = ortho % n_levels # Classic…How to Hash a User ID to an Experiment Bucket in Python
Deterministically map a user ID to one of N experiment buckets using MD5 hashing and modulo arithmetic.
import hashlib
def hash_to_bucket(user_id: str, num_buckets: int = 10) -> int:
"""Deterministically map a user_id to a bucket (0 to num_buckets-1)."""
digest = hashlib.md5(user_id.encode("utf-8")).hexdigest()
return int(digest[:8], 16) % num_buckets
if __name__ == "__main__":
# Mock experiment: split…Browse by section
Each section groups closely related Python snippets.
A/B testing & experimentation — Python code examples
What you will find here
This page collects a/b testing & experimentation snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.