A/B testing & experimentation
User bucketing, experiment metrics, statistical comparison, and rollout guardrails.
Bayesian A/B Test Credible Interval in Python
Simulates A/B test data and computes posterior credible intervals and the probability that variant B outperforms A using Bayesian Beta-Binomial inference.
import numpy as np
from scipy import stats
# Simulated A/B test data
n_A = 1000
n_B = 1000
conversions_A = 120
conversions_B = 140
# Prior: Beta(1, 1) uniform
alpha_prior, beta_prior = 1, 1
# Posterior parameters
alpha_A = alpha_prior + conversions_A
beta_A = beta_prior + n_A - conversions_A
alpha_B = alpha_prior +…Benjamini Hochberg FDR Correction in Python
Implement the Benjamini-HHochberg false discovery rate (FDR) procedure in Python to control the expected proportion of false positives among rejected hypotheses.
import numpy as np
def benjamini_hochberg(p_values, alpha=0.05):
p_values = np.array(p_values)
n = len(p_values)
sorted_idx = np.argsort(p_values)
sorted_p = p_values[sorted_idx]
thresholds = (np.arange(1, n + 1) / n) * alpha
significant = sorted_p <= thresholds
if not significan…Bonferroni Correction in Python
Applies the Bonferroni correction to a list of p-values to control the family-wise error rate when performing multiple comparisons.
import numpy as np
def bonferroni_correction(p_values, alpha=0.05):
"""Apply Bonferroni correction to a list of p-values."""
n = len(p_values)
corrected_alpha = alpha / n
significant = [p < corrected_alpha for p in p_values]
return corrected_alpha, significant
if __name__ == "__main__":
# Moc…Bootstrap Confidence Interval in Python
Estimates a confidence interval for a statistic (like the mean) using bootstrap resampling in pure Python.
import random
def bootstrap_ci(data, statistic, n_bootstraps=1000, ci_level=0.95, seed=42):
random.seed(seed)
n = len(data)
boot_stats = []
for _ in range(n_bootstraps):
sample = [random.choice(data) for _ in range(n)]
boot_stats.append(statistic(sample))
boot_stats.sort()
l…Check Sample Ratio Mismatch in Python
Estimates the probability that a simple random sample's proportion differs from the population proportion by more than 10% using simulation.
import random
def sample_ratio_mismatch(population_size: int, sample_size: int, p: float) -> float:
"""
Estimate the probability that a simple random sample's proportion
differs from the population proportion by more than 10%.
"""
total_counts = [0, 0]
for _ in range(10000):
sample = …Chi-Square Test in Python for Conversion Mock Data
Compute the chi-square statistic and approximate p-value for a mock A/B conversion test using the standard library.
import math
from collections import Counter
def chi_square_statistic(observed):
"""
Compute chi-square statistic for a mock conversion test.
observed: dict mapping outcomes to observed frequencies.
"""
observed = Counter(observed)
n = sum(observed.values())
expected = n / len(observed) if …Delta Method for Ratio Metrics in A/B Testing with Python
Computes the confidence interval for the difference between two ratio metrics using the delta method, with mock A/B test data.
import numpy as np
from scipy.stats import norm
def delta_method_ratio_delta(control: np.ndarray, treatment: np.ndarray, confidence: float = 0.95):
"""Estimate confidence interval for ratio metric using delta method.
Args:
control: numerator/denominator pairs from control group (n x 2 array)
…Generate a Mock Multi-Armed Bandit Report in Python
Simulate a multi-armed bandit experiment with random pulls and rewards, then output a JSON report with per-arm statistics.
import random
import json
def generate_mock_bandit_report(num_arms=5, num_rounds=100, seed=42):
random.seed(seed)
arms = ["A", "B", "C", "D", "E"][:num_arms]
true_means = {arm: random.uniform(0.3, 0.7) for arm in arms}
pulls = {arm: 0 for arm in arms}
rewards = {arm: 0 for arm in arms}
for _ …How to Build a Guardrail Metrics Monitor in Python
This code implements a mock monitor that records metric values, checks them against thresholds, and summarizes pass/alert statistics.
import random
import time
from collections import defaultdict
class GuardrailMetricsMonitor:
def __init__(self):
self.metrics = defaultdict(list)
self.thresholds = {
"prompt_toxicity": 0.8,
"response_length": 500,
"latency_ms": 1000,
}
def record(s…How to Calculate Minimum Sample Size for a T-Test in Python
Compute the minimum sample size per group for a two-sample t-test using effect size, significance level, and statistical power.
import math
from scipy.stats import norm
def min_sample_size(effect_size, alpha=0.05, power=0.8):
"""
Calculate minimum sample size for a two-sample t-test (equal groups).
Args:
effect_size: Cohen's d (standardized mean difference)
alpha: significance level (Type I error)
power: …How to Calculate Secondary Metrics in Python
Computes distribution, variability, and spread of a numeric dataset using Python's statistics and collections modules.
import random
import statistics
from collections import Counter
def explore_secondary_metrics(data):
"""Calculate secondary metrics: distribution, variability, and spread."""
if not data:
return "No data provided"
total = sum(data)
mean = statistics.mean(data)
median = statistics.medi…How to Calculate Weighted Grades and Generate Mock Notes in Python
Compute a weighted physics grade from exam and homework scores, then generate a performance-based mock note with percentage and feedback.
def get_physics_grade(exam_score, homework_score):
"""Calculate final grade from exam and homework scores."""
exam_weight = 0.7
homework_weight = 0.3
return (exam_score * exam_weight) + (homework_score * homework_weight)
def mock_note(correct_score, max_score, student_name):
"""Generate a mock no…How to Compute Mann-Whitney U Test in Python
Compute the Mann-Whitney U statistic and p-value manually in Python with tie correction and a normal approximation for independent samples.
import numpy as np
from scipy import stats
def mann_whitney_u_mock(sample_a, sample_b):
"""Compute Mann-Whitney U and p-value manually."""
# Combine and rank
combined = sample_a + sample_b
n_a, n_b = len(sample_a), len(sample_b)
n_total = n_a + n_b
# Rank with ties handling (average ranks…How to Conduct a Two-Sample T-Test in Python
Performs Welch's t-test for two independent samples, computing the t-statistic, degrees of freedom, and p-value using NumPy and SciPy.
import numpy as np
def two_sample_t_test(sample1, sample2):
"""Perform Welch's t-test for two independent samples."""
n1, n2 = len(sample1), len(sample2)
mean1, mean2 = np.mean(sample1), np.mean(sample2)
var1, var2 = np.var(sample1, ddof=1), np.var(sample2, ddof=1)
# Standard error of difference
…How to Define a Mock Primary Metric in Python
Define a mock primary metric object with a name, value, and unit, and serialize it to a dictionary for experimentation and testing.
class Metric:
def __init__(self, name, value, unit=None):
self.name = name
self.value = value
self.unit = unit
def to_dict(self):
result = {"name": self.name, "value": self.value}
if self.unit:
result["unit"] = self.unit
return result
def __repr…How to Mock a Confidence Interval for a Proportion in Python
Simulate a Bernoulli sample and compute a 95% confidence interval for a proportion using the normal approximation in Python.
import random
import math
def mock_ci(n=100, p_true=0.5, z=1.96, seed=42):
"""Simulate a sample proportion and compute its 95% confidence interval."""
random.seed(seed)
successes = sum(1 for _ in range(n) if random.random() < p_true)
p_hat = successes / n
se = math.sqrt(p_hat * (1 - p_hat) / n)
…How to Perform Intent-to-Treat Analysis in Python
Runs an intent-to-treat analysis on mock A/B test data, comparing outcomes by initial group assignment with a t-test for significance.
import pandas as pd
import numpy as np
def intent_to_treat_analysis(data):
"""Perform intent-to-treat (ITT) analysis.
ITT compares outcomes based on initial treatment assignment,
regardless of whether participants actually received the treatment.
"""
# Create a copy to avoid mutating the origina…How to Perform Welch's t-Test in Python
Calculate the Welch t-statistic and degrees of freedom for two samples with unequal variances using Python's statistics module.
import math
from statistics import mean, variance
def welch_t_test(sample1, sample2):
n1, n2 = len(sample1), len(sample2)
mean1, mean2 = mean(sample1), mean(sample2)
var1, var2 = variance(sample1), variance(sample2)
# Welch's t statistic
t_stat = (mean1 - mean2) / math.sqrt(var1 / n1 + var2 / n2…How to Run a Fisher Exact Test in Python
Compute the two-sided Fisher exact test p-value for a 2x2 contingency table using pure Python and the math module.
from math import comb, factorial
from itertools import combinations
def hypergeometric_probability(a, b, c, d):
"""Probability of observing table [[a, b], [c, d]] under the null."""
row1 = a + b
row2 = c + d
col1 = a + c
col2 = b + d
total = row1 + row2
return (comb(row1, a) * comb(row2, …How to Run a Permutation Test in Python
Run a Monte Carlo permutation test to compute a p-value for comparing two group means without parametric assumptions.
import random
import statistics
def permutation_test(group_a, group_b, n_permutations=10000, seed=42):
random.seed(seed)
combined = group_a + group_b
observed_diff = abs(statistics.mean(group_a) - statistics.mean(group_b))
count = 0
n = len(group_a)
for _ in range(n_permutations):
…How to Simulate Geo Experiments in Python
Build a mock geo experiment simulator with ramp-up/down periods, measuring weekly lift between treatment and control markets.
import random
import math
from dataclasses import dataclass
@dataclass
class GeoMarket:
name: str
base_demand: float
geo_coefficient: float
def simulate_geo_experiment(markets, weeks=12, control_weeks=6):
"""
Simulates a geo experiment with ramp-up and ramp-down periods.
Returns weekly lift p…Browse by section
Each section groups closely related Python snippets.
A/B testing & experimentation — Python code examples
What you will find here
This page collects a/b testing & experimentation snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.