A/B testing & experimentation
User bucketing, experiment metrics, statistical comparison, and rollout guardrails.
Bayesian A/B Test Credible Interval in Python
Simulates A/B test data and computes posterior credible intervals and the probability that variant B outperforms A using Bayesian Beta-Binomial inference.
import numpy as np
from scipy import stats
# Simulated A/B test data
n_A = 1000
n_B = 1000
conversions_A = 120
conversions_B = 140
# Prior: Beta(1, 1) uniform
alpha_prior, beta_prior = 1, 1
# Posterior parameters
alpha_A = alpha_prior + conversions_A
beta_A = beta_prior + n_A - conversions_A
alpha_B = alpha_prior +…Benjamini Hochberg FDR Correction in Python
Implement the Benjamini-HHochberg false discovery rate (FDR) procedure in Python to control the expected proportion of false positives among rejected hypotheses.
import numpy as np
def benjamini_hochberg(p_values, alpha=0.05):
p_values = np.array(p_values)
n = len(p_values)
sorted_idx = np.argsort(p_values)
sorted_p = p_values[sorted_idx]
thresholds = (np.arange(1, n + 1) / n) * alpha
significant = sorted_p <= thresholds
if not significan…Bonferroni Correction in Python
Applies the Bonferroni correction to a list of p-values to control the family-wise error rate when performing multiple comparisons.
import numpy as np
def bonferroni_correction(p_values, alpha=0.05):
"""Apply Bonferroni correction to a list of p-values."""
n = len(p_values)
corrected_alpha = alpha / n
significant = [p < corrected_alpha for p in p_values]
return corrected_alpha, significant
if __name__ == "__main__":
# Moc…Check Sample Ratio Mismatch in Python
Estimates the probability that a simple random sample's proportion differs from the population proportion by more than 10% using simulation.
import random
def sample_ratio_mismatch(population_size: int, sample_size: int, p: float) -> float:
"""
Estimate the probability that a simple random sample's proportion
differs from the population proportion by more than 10%.
"""
total_counts = [0, 0]
for _ in range(10000):
sample = …Chi-Square Test in Python for Conversion Mock Data
Compute the chi-square statistic and approximate p-value for a mock A/B conversion test using the standard library.
import math
from collections import Counter
def chi_square_statistic(observed):
"""
Compute chi-square statistic for a mock conversion test.
observed: dict mapping outcomes to observed frequencies.
"""
observed = Counter(observed)
n = sum(observed.values())
expected = n / len(observed) if …Delta Method for Ratio Metrics in A/B Testing with Python
Computes the confidence interval for the difference between two ratio metrics using the delta method, with mock A/B test data.
import numpy as np
from scipy.stats import norm
def delta_method_ratio_delta(control: np.ndarray, treatment: np.ndarray, confidence: float = 0.95):
"""Estimate confidence interval for ratio metric using delta method.
Args:
control: numerator/denominator pairs from control group (n x 2 array)
…Difference in Differences Mock in Python
Generate mock panel data with a known treatment effect and compute a difference-in-differences estimate using group and period means.
import numpy as np
import pandas as pd
# Generate mock panel data: 2 groups (control=0, treatment=1) × 2 periods (pre=0, post=1)
rng = np.random.default_rng(42)
n_per_cell = 50
data = []
for group in [0, 1]:
for period in [0, 1]:
# True effect: treatment increases outcome by 5 in the post period
…Epsilon Greedy Bandit Mock in Python
A simple epsilon-greedy multi-armed bandit simulation that balances exploration and exploitation to estimate true means of several Bernoulli-like reward distributions.
import random
class Bandit:
def __init__(self, true_mean):
self.true_mean = true_mean
self.estimated_mean = 0.0
self.n_pulls = 0
def pull(self):
return random.gauss(self.true_mean, 1.0)
def update(self, reward):
self.n_pulls += 1
self.estimated_mean += (r…Generate a Mock Multi-Armed Bandit Report in Python
Simulate a multi-armed bandit experiment with random pulls and rewards, then output a JSON report with per-arm statistics.
import random
import json
def generate_mock_bandit_report(num_arms=5, num_rounds=100, seed=42):
random.seed(seed)
arms = ["A", "B", "C", "D", "E"][:num_arms]
true_means = {arm: random.uniform(0.3, 0.7) for arm in arms}
pulls = {arm: 0 for arm in arms}
rewards = {arm: 0 for arm in arms}
for _ …How to Build a Simple Binary Protocol Parser Mock in Python
Defines a mock binary protocol with field definitions, encoding, and decoding to simulate network packet parsing for A/B testing and experiment setup.
class SimpleProtocol:
def __init__(self, name, version):
self.name = name
self.version = version
self.fields = []
def add_field(self, field_name, field_size):
self.fields.append((field_name, field_size))
def parse(self, data):
if len(data) != sum(size for _, size i…How to Calculate Secondary Metrics in Python
Computes distribution, variability, and spread of a numeric dataset using Python's statistics and collections modules.
import random
import statistics
from collections import Counter
def explore_secondary_metrics(data):
"""Calculate secondary metrics: distribution, variability, and spread."""
if not data:
return "No data provided"
total = sum(data)
mean = statistics.mean(data)
median = statistics.medi…How to Calculate Weighted Grades and Generate Mock Notes in Python
Compute a weighted physics grade from exam and homework scores, then generate a performance-based mock note with percentage and feedback.
def get_physics_grade(exam_score, homework_score):
"""Calculate final grade from exam and homework scores."""
exam_weight = 0.7
homework_weight = 0.3
return (exam_score * exam_weight) + (homework_score * homework_weight)
def mock_note(correct_score, max_score, student_name):
"""Generate a mock no…How to Compute CUPED Variance Reduction in Python
Implement CUPED in Python to reduce variance of A/B test treatment effect estimates using pre-experiment covariates.
import numpy as np
def compute_cuped_reduction(control, variant, covariate):
"""
Compute variance reduction using CUPED (Controlled Experiment with
Pre-Experiment Data). Uses pre-experiment covariate values to
reduce variance of the treatment effect estimate.
"""
control = np.asarray(control, …How to Compute Mann-Whitney U Test in Python
Compute the Mann-Whitney U statistic and p-value manually in Python with tie correction and a normal approximation for independent samples.
import numpy as np
from scipy import stats
def mann_whitney_u_mock(sample_a, sample_b):
"""Compute Mann-Whitney U and p-value manually."""
# Combine and rank
combined = sample_a + sample_b
n_a, n_b = len(sample_a), len(sample_b)
n_total = n_a + n_b
# Rank with ties handling (average ranks…How to Create an Interrupted Time Series Mock in Python
Generate simulated interrupted time series data with a pre/post-intervention trend, level shift, and noise to test segmented regression models.
import numpy as np
# Mock interrupted time series data
np.random.seed(42)
n_pre = 50
n_post = 50
time = np.arange(0, n_pre + n_post)
# Pre-intervention: linear trend + noise
pre_trend = 0.05 * time[:n_pre] + np.random.normal(0, 0.5, n_pre)
# Post-intervention: new slope + level shift + noise
post_trend = 0.05 * tim…How to Define a Mock Primary Metric in Python
Define a mock primary metric object with a name, value, and unit, and serialize it to a dictionary for experimentation and testing.
class Metric:
def __init__(self, name, value, unit=None):
self.name = name
self.value = value
self.unit = unit
def to_dict(self):
result = {"name": self.name, "value": self.value}
if self.unit:
result["unit"] = self.unit
return result
def __repr…How to Do Random Assignment in Python for A/B Tests
Assign each item to a binary group (0 or 1) with uniform probability using a small reusable function, optionally weighted, for A/B testing mocks.
import random
def random_assignment_uniform_mock(items, weights=None):
"""Assign each item to a group (0 or 1) with uniform probability."""
if weights is None:
# Default: each item independently gets 0 or 1 with 50% probability
return [random.randint(0, 1) for _ in items]
# Optional weight…How to Evaluate Feature Flags in Python
A Python function that evaluates boolean feature flags with user-specific overrides, returning whether a flag is enabled and the reason for the decision.
import json
def evaluate_feature_flag(feature_name, context, flag_configs):
"""
Evaluates a boolean feature flag given a context dictionary.
Args:
feature_name: The name of the feature flag.
context: A dictionary of user/request context (e.g., {"user_id": "123"}).
flag_configs: A …How to Generate Multivariate JSON Mock Data in Python
This script generates mock multivariate JSON-compatible data with measurements and boolean flags for testing and experimentation pipelines.
import json
def multivariate_mock(row_count: int = 3) -> list:
"""Generate mock multivariate data as list of JSON-compatible dicts."""
records = []
for i in range(row_count):
record = {
"id": i + 1,
"measurements": {
"temperature": 20.5 + i * 1.5,
…How to Generate an Orthogonal Array for A/B Testing in Python
Generate a mock orthogonal array for multi-layer experiments with NumPy, ensuring balanced level combinations across experiment groups.
import numpy as np
def orthogonal_mock_layers(n_experiments: int, n_layers: int, n_levels: int) -> np.ndarray:
"""Generate an orthogonal array for multi-layer experiment design using base-level logic."""
ortho = np.indices((n_levels,) * n_layers).reshape(n_layers, -1).T
ortho = ortho % n_levels # Classic…How to Mock Mutual Exclusion for A/B Experiment Groups in Python
Simulate mutual exclusion for experiment groups using a thread-safe lock, ensuring only one member updates the shared counter at a time.
import threading
import time
import random
class CountingGate:
"""A mock mutual exclusion gate using a lock."""
def __init__(self):
self.counter = 0
self.lock = threading.Lock()
def enter(self, group_id, member_id):
with self.lock:
current = self.counter
t…How to Mock Stratified Assignment by Segment in Python
Simulate stratified assignment for A/B experiments by sampling a fixed proportion of units from each segment, with deterministic seeds for reproducibility.
import random
def stratified_assignment(segments, seed=None):
"""
Mock stratified assignment: given a dict of segment -> population size,
return a dict of segment -> sampled unit ids (deterministic with seed).
"""
if seed is not None:
random.seed(seed)
rng = random.Random(seed)
res…How to Mock Time for Cache TTL Testing in Python
This code demonstrates how to test a cache's TTL expiration logic by mocking time.time with unittest.mock to control the passage of time.
import time
from unittest.mock import patch
class ConfigCache:
def __init__(self, ttl=60):
self.ttl = ttl
self._store = {}
self._timestamps = {}
def get(self, key):
if key not in self._store:
return None
if time.time() - self._timestamps[key] > self.ttl:
…How to Mock a Confidence Interval for a Proportion in Python
Simulate a Bernoulli sample and compute a 95% confidence interval for a proportion using the normal approximation in Python.
import random
import math
def mock_ci(n=100, p_true=0.5, z=1.96, seed=42):
"""Simulate a sample proportion and compute its 95% confidence interval."""
random.seed(seed)
successes = sum(1 for _ in range(n) if random.random() < p_true)
p_hat = successes / n
se = math.sqrt(p_hat * (1 - p_hat) / n)
…Browse by section
Each section groups closely related Python snippets.
A/B testing & experimentation — Python code examples
What you will find here
This page collects a/b testing & experimentation snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.