A/B testing & experimentation
User bucketing, experiment metrics, statistical comparison, and rollout guardrails.
How to Compute CUPED Variance Reduction in Python
Implement CUPED in Python to reduce variance of A/B test treatment effect estimates using pre-experiment covariates.
import numpy as np
def compute_cuped_reduction(control, variant, covariate):
"""
Compute variance reduction using CUPED (Controlled Experiment with
Pre-Experiment Data). Uses pre-experiment covariate values to
reduce variance of the treatment effect estimate.
"""
control = np.asarray(control, …How to Compute Mann-Whitney U Test in Python
Compute the Mann-Whitney U statistic and p-value manually in Python with tie correction and a normal approximation for independent samples.
import numpy as np
from scipy import stats
def mann_whitney_u_mock(sample_a, sample_b):
"""Compute Mann-Whitney U and p-value manually."""
# Combine and rank
combined = sample_a + sample_b
n_a, n_b = len(sample_a), len(sample_b)
n_total = n_a + n_b
# Rank with ties handling (average ranks…How to Conduct a Two-Sample T-Test in Python
Performs Welch's t-test for two independent samples, computing the t-statistic, degrees of freedom, and p-value using NumPy and SciPy.
import numpy as np
def two_sample_t_test(sample1, sample2):
"""Perform Welch's t-test for two independent samples."""
n1, n2 = len(sample1), len(sample2)
mean1, mean2 = np.mean(sample1), np.mean(sample2)
var1, var2 = np.var(sample1, ddof=1), np.var(sample2, ddof=1)
# Standard error of difference
…How to Create an Interrupted Time Series Mock in Python
Generate simulated interrupted time series data with a pre/post-intervention trend, level shift, and noise to test segmented regression models.
import numpy as np
# Mock interrupted time series data
np.random.seed(42)
n_pre = 50
n_post = 50
time = np.arange(0, n_pre + n_post)
# Pre-intervention: linear trend + noise
pre_trend = 0.05 * time[:n_pre] + np.random.normal(0, 0.5, n_pre)
# Post-intervention: new slope + level shift + noise
post_trend = 0.05 * tim…How to Generate an Orthogonal Array for A/B Testing in Python
Generate a mock orthogonal array for multi-layer experiments with NumPy, ensuring balanced level combinations across experiment groups.
import numpy as np
def orthogonal_mock_layers(n_experiments: int, n_layers: int, n_levels: int) -> np.ndarray:
"""Generate an orthogonal array for multi-layer experiment design using base-level logic."""
ortho = np.indices((n_levels,) * n_layers).reshape(n_layers, -1).T
ortho = ortho % n_levels # Classic…How to Mock Mutual Exclusion for A/B Experiment Groups in Python
Simulate mutual exclusion for experiment groups using a thread-safe lock, ensuring only one member updates the shared counter at a time.
import threading
import time
import random
class CountingGate:
"""A mock mutual exclusion gate using a lock."""
def __init__(self):
self.counter = 0
self.lock = threading.Lock()
def enter(self, group_id, member_id):
with self.lock:
current = self.counter
t…How to Mock Sequential Calls in Python with unittest.mock
Use Mock.side_effect to return a different result for each sequential call and verify the call order with assert_has_calls.
import unittest
from unittest.mock import Mock
class Service:
def fetch(self, item_id):
raise NotImplementedError
def process_items(service, ids):
results = []
for item_id in ids:
result = service.fetch(item_id)
results.append(result)
return results
if __name__ == "__main__":…How to Mock Time for Cache TTL Testing in Python
This code demonstrates how to test a cache's TTL expiration logic by mocking time.time with unittest.mock to control the passage of time.
import time
from unittest.mock import patch
class ConfigCache:
def __init__(self, ttl=60):
self.ttl = ttl
self._store = {}
self._timestamps = {}
def get(self, key):
if key not in self._store:
return None
if time.time() - self._timestamps[key] > self.ttl:
…How to Perform Intent-to-Treat Analysis in Python
Runs an intent-to-treat analysis on mock A/B test data, comparing outcomes by initial group assignment with a t-test for significance.
import pandas as pd
import numpy as np
def intent_to_treat_analysis(data):
"""Perform intent-to-treat (ITT) analysis.
ITT compares outcomes based on initial treatment assignment,
regardless of whether participants actually received the treatment.
"""
# Create a copy to avoid mutating the origina…How to Perform Welch's t-Test in Python
Calculate the Welch t-statistic and degrees of freedom for two samples with unequal variances using Python's statistics module.
import math
from statistics import mean, variance
def welch_t_test(sample1, sample2):
n1, n2 = len(sample1), len(sample2)
mean1, mean2 = mean(sample1), mean(sample2)
var1, var2 = variance(sample1), variance(sample2)
# Welch's t statistic
t_stat = (mean1 - mean2) / math.sqrt(var1 / n1 + var2 / n2…How to Run a Fisher Exact Test in Python
Compute the two-sided Fisher exact test p-value for a 2x2 contingency table using pure Python and the math module.
from math import comb, factorial
from itertools import combinations
def hypergeometric_probability(a, b, c, d):
"""Probability of observing table [[a, b], [c, d]] under the null."""
row1 = a + b
row2 = c + d
col1 = a + c
col2 = b + d
total = row1 + row2
return (comb(row1, a) * comb(row2, …How to Run a Permutation Test in Python
Run a Monte Carlo permutation test to compute a p-value for comparing two group means without parametric assumptions.
import random
import statistics
def permutation_test(group_a, group_b, n_permutations=10000, seed=42):
random.seed(seed)
combined = group_a + group_b
observed_diff = abs(statistics.mean(group_a) - statistics.mean(group_b))
count = 0
n = len(group_a)
for _ in range(n_permutations):
…How to Simulate Geo Experiments in Python
Build a mock geo experiment simulator with ramp-up/down periods, measuring weekly lift between treatment and control markets.
import random
import math
from dataclasses import dataclass
@dataclass
class GeoMarket:
name: str
base_demand: float
geo_coefficient: float
def simulate_geo_experiment(markets, weeks=12, control_weeks=6):
"""
Simulates a geo experiment with ramp-up and ramp-down periods.
Returns weekly lift p…How to join assignment logs with outcomes in Python
Merge submission log entries with grading outcomes using left join and full outer join patterns in pure Python.
from datetime import datetime, timedelta
class AssignmentLog:
def __init__(self):
self.logs = [
{"assignment_id": 101, "student_id": "S001", "submitted_at": "2024-03-01 10:30:00"},
{"assignment_id": 101, "student_id": "S002", "submitted_at": "2024-03-02 14:15:00"},
{"as…How to simulate a contextual bandit in Python
Simulate a contextual multi-armed bandit with random features and epsilon-greedy action selection in Python.
import random
class ContextualBandit:
def __init__(self, n_actions=3, n_features=4):
self.n_actions = n_actions
self.n_features = n_features
self.theta = [random.random() for _ in range(n_actions * n_features)]
def mock_context(self):
return [random.uniform(-1, 1) for _ in ra…Browse by section
Each section groups closely related Python snippets.
A/B testing & experimentation — Python code examples
What you will find here
This page collects a/b testing & experimentation snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.