A/B testing & experimentation
User bucketing, experiment metrics, statistical comparison, and rollout guardrails.
How to Build a Guardrail Metrics Monitor in Python
This code implements a mock monitor that records metric values, checks them against thresholds, and summarizes pass/alert statistics.
import random
import time
from collections import defaultdict
class GuardrailMetricsMonitor:
def __init__(self):
self.metrics = defaultdict(list)
self.thresholds = {
"prompt_toxicity": 0.8,
"response_length": 500,
"latency_ms": 1000,
}
def record(s…How to Build a Simple Binary Protocol Parser Mock in Python
Defines a mock binary protocol with field definitions, encoding, and decoding to simulate network packet parsing for A/B testing and experiment setup.
class SimpleProtocol:
def __init__(self, name, version):
self.name = name
self.version = version
self.fields = []
def add_field(self, field_name, field_size):
self.fields.append((field_name, field_size))
def parse(self, data):
if len(data) != sum(size for _, size i…How to Calculate Minimum Sample Size for a T-Test in Python
Compute the minimum sample size per group for a two-sample t-test using effect size, significance level, and statistical power.
import math
from scipy.stats import norm
def min_sample_size(effect_size, alpha=0.05, power=0.8):
"""
Calculate minimum sample size for a two-sample t-test (equal groups).
Args:
effect_size: Cohen's d (standardized mean difference)
alpha: significance level (Type I error)
power: …How to Calculate Secondary Metrics in Python
Computes distribution, variability, and spread of a numeric dataset using Python's statistics and collections modules.
import random
import statistics
from collections import Counter
def explore_secondary_metrics(data):
"""Calculate secondary metrics: distribution, variability, and spread."""
if not data:
return "No data provided"
total = sum(data)
mean = statistics.mean(data)
median = statistics.medi…How to Calculate Weighted Grades and Generate Mock Notes in Python
Compute a weighted physics grade from exam and homework scores, then generate a performance-based mock note with percentage and feedback.
def get_physics_grade(exam_score, homework_score):
"""Calculate final grade from exam and homework scores."""
exam_weight = 0.7
homework_weight = 0.3
return (exam_score * exam_weight) + (homework_score * homework_weight)
def mock_note(correct_score, max_score, student_name):
"""Generate a mock no…How to Compute CUPED Variance Reduction in Python
Implement CUPED in Python to reduce variance of A/B test treatment effect estimates using pre-experiment covariates.
import numpy as np
def compute_cuped_reduction(control, variant, covariate):
"""
Compute variance reduction using CUPED (Controlled Experiment with
Pre-Experiment Data). Uses pre-experiment covariate values to
reduce variance of the treatment effect estimate.
"""
control = np.asarray(control, …How to Compute Mann-Whitney U Test in Python
Compute the Mann-Whitney U statistic and p-value manually in Python with tie correction and a normal approximation for independent samples.
import numpy as np
from scipy import stats
def mann_whitney_u_mock(sample_a, sample_b):
"""Compute Mann-Whitney U and p-value manually."""
# Combine and rank
combined = sample_a + sample_b
n_a, n_b = len(sample_a), len(sample_b)
n_total = n_a + n_b
# Rank with ties handling (average ranks…How to Conduct a Two-Sample T-Test in Python
Performs Welch's t-test for two independent samples, computing the t-statistic, degrees of freedom, and p-value using NumPy and SciPy.
import numpy as np
def two_sample_t_test(sample1, sample2):
"""Perform Welch's t-test for two independent samples."""
n1, n2 = len(sample1), len(sample2)
mean1, mean2 = np.mean(sample1), np.mean(sample2)
var1, var2 = np.var(sample1, ddof=1), np.var(sample2, ddof=1)
# Standard error of difference
…How to Create a Mock That Returns Inverse Counter Values in Python
Builds a Mock whose side_effect returns the inverse (1/count) of each Counter value, defaulting to 0.0 for unseen keys.
from collections import Counter
from unittest.mock import Mock
def inverse_mock(counter: Counter) -> Mock:
"""
Return a Mock that mimics the inverse of a Counter:
each key returns a value representing the inverse of its count.
The Mock's side_effect maps keys to their inverse counts.
"""
mock …How to Create a Sticky Consistent Mock with unittest.mock in Python
Shows how to use unittest.mock.patch.object to mock a method consistently across multiple calls, returning a sticky value every time.
from unittest.mock import patch
class Database:
def fetch(self, key):
return f"real value for {key}"
def get_value(db, key):
return db.fetch(key)
if __name__ == "__main__":
db = Database()
with patch.object(db, "fetch", return_value="sticky value") as mock_fetch:
result1 = get_value(…How to Create an Interrupted Time Series Mock in Python
Generate simulated interrupted time series data with a pre/post-intervention trend, level shift, and noise to test segmented regression models.
import numpy as np
# Mock interrupted time series data
np.random.seed(42)
n_pre = 50
n_post = 50
time = np.arange(0, n_pre + n_post)
# Pre-intervention: linear trend + noise
pre_trend = 0.05 * time[:n_pre] + np.random.normal(0, 0.5, n_pre)
# Post-intervention: new slope + level shift + noise
post_trend = 0.05 * tim…How to Define a Mock Primary Metric in Python
Define a mock primary metric object with a name, value, and unit, and serialize it to a dictionary for experimentation and testing.
class Metric:
def __init__(self, name, value, unit=None):
self.name = name
self.value = value
self.unit = unit
def to_dict(self):
result = {"name": self.name, "value": self.value}
if self.unit:
result["unit"] = self.unit
return result
def __repr…How to Do Random Assignment in Python for A/B Tests
Assign each item to a binary group (0 or 1) with uniform probability using a small reusable function, optionally weighted, for A/B testing mocks.
import random
def random_assignment_uniform_mock(items, weights=None):
"""Assign each item to a group (0 or 1) with uniform probability."""
if weights is None:
# Default: each item independently gets 0 or 1 with 50% probability
return [random.randint(0, 1) for _ in items]
# Optional weight…How to Evaluate Feature Flags in Python
A Python function that evaluates boolean feature flags with user-specific overrides, returning whether a flag is enabled and the reason for the decision.
import json
def evaluate_feature_flag(feature_name, context, flag_configs):
"""
Evaluates a boolean feature flag given a context dictionary.
Args:
feature_name: The name of the feature flag.
context: A dictionary of user/request context (e.g., {"user_id": "123"}).
flag_configs: A …How to Generate Multivariate JSON Mock Data in Python
This script generates mock multivariate JSON-compatible data with measurements and boolean flags for testing and experimentation pipelines.
import json
def multivariate_mock(row_count: int = 3) -> list:
"""Generate mock multivariate data as list of JSON-compatible dicts."""
records = []
for i in range(row_count):
record = {
"id": i + 1,
"measurements": {
"temperature": 20.5 + i * 1.5,
…How to Generate an Orthogonal Array for A/B Testing in Python
Generate a mock orthogonal array for multi-layer experiments with NumPy, ensuring balanced level combinations across experiment groups.
import numpy as np
def orthogonal_mock_layers(n_experiments: int, n_layers: int, n_levels: int) -> np.ndarray:
"""Generate an orthogonal array for multi-layer experiment design using base-level logic."""
ortho = np.indices((n_levels,) * n_layers).reshape(n_layers, -1).T
ortho = ortho % n_levels # Classic…How to Hash a User ID to an Experiment Bucket in Python
Deterministically map a user ID to one of N experiment buckets using MD5 hashing and modulo arithmetic.
import hashlib
def hash_to_bucket(user_id: str, num_buckets: int = 10) -> int:
"""Deterministically map a user_id to a bucket (0 to num_buckets-1)."""
digest = hashlib.md5(user_id.encode("utf-8")).hexdigest()
return int(digest[:8], 16) % num_buckets
if __name__ == "__main__":
# Mock experiment: split…How to Mock Mutual Exclusion for A/B Experiment Groups in Python
Simulate mutual exclusion for experiment groups using a thread-safe lock, ensuring only one member updates the shared counter at a time.
import threading
import time
import random
class CountingGate:
"""A mock mutual exclusion gate using a lock."""
def __init__(self):
self.counter = 0
self.lock = threading.Lock()
def enter(self, group_id, member_id):
with self.lock:
current = self.counter
t…How to Mock Sequential Calls in Python with unittest.mock
Use Mock.side_effect to return a different result for each sequential call and verify the call order with assert_has_calls.
import unittest
from unittest.mock import Mock
class Service:
def fetch(self, item_id):
raise NotImplementedError
def process_items(service, ids):
results = []
for item_id in ids:
result = service.fetch(item_id)
results.append(result)
return results
if __name__ == "__main__":…How to Mock Stratified Assignment by Segment in Python
Simulate stratified assignment for A/B experiments by sampling a fixed proportion of units from each segment, with deterministic seeds for reproducibility.
import random
def stratified_assignment(segments, seed=None):
"""
Mock stratified assignment: given a dict of segment -> population size,
return a dict of segment -> sampled unit ids (deterministic with seed).
"""
if seed is not None:
random.seed(seed)
rng = random.Random(seed)
res…How to Mock Time for Cache TTL Testing in Python
This code demonstrates how to test a cache's TTL expiration logic by mocking time.time with unittest.mock to control the passage of time.
import time
from unittest.mock import patch
class ConfigCache:
def __init__(self, ttl=60):
self.ttl = ttl
self._store = {}
self._timestamps = {}
def get(self, key):
if key not in self._store:
return None
if time.time() - self._timestamps[key] > self.ttl:
…How to Mock a Confidence Interval for a Proportion in Python
Simulate a Bernoulli sample and compute a 95% confidence interval for a proportion using the normal approximation in Python.
import random
import math
def mock_ci(n=100, p_true=0.5, z=1.96, seed=42):
"""Simulate a sample proportion and compute its 95% confidence interval."""
random.seed(seed)
successes = sum(1 for _ in range(n) if random.random() < p_true)
p_hat = successes / n
se = math.sqrt(p_hat * (1 - p_hat) / n)
…How to Mock a Remote Config Fetch in Python
Simulate a remote config API response with metadata, timestamps, and mock data for testing or local development.
import json
from datetime import datetime
from typing import Any, Dict
def fetch_remote_config(mock_data: Dict[str, Any]) -> Dict[str, Any]:
"""Simulate fetching a remote config with metadata and timestamps."""
return {
"status": "success",
"source": "mock",
"fetched_at": datetime.utcn…How to Mock an Exposure Event Log Record in Python
Generate a realistic exposure event record with UUID, UTC timestamp, and risk level for testing or experimentation.
import uuid
from datetime import datetime, timezone
def mock_exposure_event(person_id: str, location: str, duration_minutes: int) -> dict:
return {
"event_id": str(uuid.uuid4()),
"person_id": person_id,
"location": location,
"duration_minutes": duration_minutes,
"timestamp…Browse by section
Each section groups closely related Python snippets.
A/B testing & experimentation — Python code examples
What you will find here
This page collects a/b testing & experimentation snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.