A/B testing & experimentation
User bucketing, experiment metrics, statistical comparison, and rollout guardrails.
How to Evaluate Feature Flags in Python
A Python function that evaluates boolean feature flags with user-specific overrides, returning whether a flag is enabled and the reason for the decision.
import json
def evaluate_feature_flag(feature_name, context, flag_configs):
"""
Evaluates a boolean feature flag given a context dictionary.
Args:
feature_name: The name of the feature flag.
context: A dictionary of user/request context (e.g., {"user_id": "123"}).
flag_configs: A …How to Generate Multivariate JSON Mock Data in Python
This script generates mock multivariate JSON-compatible data with measurements and boolean flags for testing and experimentation pipelines.
import json
def multivariate_mock(row_count: int = 3) -> list:
"""Generate mock multivariate data as list of JSON-compatible dicts."""
records = []
for i in range(row_count):
record = {
"id": i + 1,
"measurements": {
"temperature": 20.5 + i * 1.5,
…How to Generate an Orthogonal Array for A/B Testing in Python
Generate a mock orthogonal array for multi-layer experiments with NumPy, ensuring balanced level combinations across experiment groups.
import numpy as np
def orthogonal_mock_layers(n_experiments: int, n_layers: int, n_levels: int) -> np.ndarray:
"""Generate an orthogonal array for multi-layer experiment design using base-level logic."""
ortho = np.indices((n_levels,) * n_layers).reshape(n_layers, -1).T
ortho = ortho % n_levels # Classic…How to Hash a User ID to an Experiment Bucket in Python
Deterministically map a user ID to one of N experiment buckets using MD5 hashing and modulo arithmetic.
import hashlib
def hash_to_bucket(user_id: str, num_buckets: int = 10) -> int:
"""Deterministically map a user_id to a bucket (0 to num_buckets-1)."""
digest = hashlib.md5(user_id.encode("utf-8")).hexdigest()
return int(digest[:8], 16) % num_buckets
if __name__ == "__main__":
# Mock experiment: split…How to Mock Mutual Exclusion for A/B Experiment Groups in Python
Simulate mutual exclusion for experiment groups using a thread-safe lock, ensuring only one member updates the shared counter at a time.
import threading
import time
import random
class CountingGate:
"""A mock mutual exclusion gate using a lock."""
def __init__(self):
self.counter = 0
self.lock = threading.Lock()
def enter(self, group_id, member_id):
with self.lock:
current = self.counter
t…How to Mock Sequential Calls in Python with unittest.mock
Use Mock.side_effect to return a different result for each sequential call and verify the call order with assert_has_calls.
import unittest
from unittest.mock import Mock
class Service:
def fetch(self, item_id):
raise NotImplementedError
def process_items(service, ids):
results = []
for item_id in ids:
result = service.fetch(item_id)
results.append(result)
return results
if __name__ == "__main__":…How to Mock Stratified Assignment by Segment in Python
Simulate stratified assignment for A/B experiments by sampling a fixed proportion of units from each segment, with deterministic seeds for reproducibility.
import random
def stratified_assignment(segments, seed=None):
"""
Mock stratified assignment: given a dict of segment -> population size,
return a dict of segment -> sampled unit ids (deterministic with seed).
"""
if seed is not None:
random.seed(seed)
rng = random.Random(seed)
res…How to Mock Time for Cache TTL Testing in Python
This code demonstrates how to test a cache's TTL expiration logic by mocking time.time with unittest.mock to control the passage of time.
import time
from unittest.mock import patch
class ConfigCache:
def __init__(self, ttl=60):
self.ttl = ttl
self._store = {}
self._timestamps = {}
def get(self, key):
if key not in self._store:
return None
if time.time() - self._timestamps[key] > self.ttl:
…How to Mock a Confidence Interval for a Proportion in Python
Simulate a Bernoulli sample and compute a 95% confidence interval for a proportion using the normal approximation in Python.
import random
import math
def mock_ci(n=100, p_true=0.5, z=1.96, seed=42):
"""Simulate a sample proportion and compute its 95% confidence interval."""
random.seed(seed)
successes = sum(1 for _ in range(n) if random.random() < p_true)
p_hat = successes / n
se = math.sqrt(p_hat * (1 - p_hat) / n)
…How to Mock a Remote Config Fetch in Python
Simulate a remote config API response with metadata, timestamps, and mock data for testing or local development.
import json
from datetime import datetime
from typing import Any, Dict
def fetch_remote_config(mock_data: Dict[str, Any]) -> Dict[str, Any]:
"""Simulate fetching a remote config with metadata and timestamps."""
return {
"status": "success",
"source": "mock",
"fetched_at": datetime.utcn…How to Mock an Exposure Event Log Record in Python
Generate a realistic exposure event record with UUID, UTC timestamp, and risk level for testing or experimentation.
import uuid
from datetime import datetime, timezone
def mock_exposure_event(person_id: str, location: str, duration_minutes: int) -> dict:
return {
"event_id": str(uuid.uuid4()),
"person_id": person_id,
"location": location,
"duration_minutes": duration_minutes,
"timestamp…How to Perform Intent-to-Treat Analysis in Python
Runs an intent-to-treat analysis on mock A/B test data, comparing outcomes by initial group assignment with a t-test for significance.
import pandas as pd
import numpy as np
def intent_to_treat_analysis(data):
"""Perform intent-to-treat (ITT) analysis.
ITT compares outcomes based on initial treatment assignment,
regardless of whether participants actually received the treatment.
"""
# Create a copy to avoid mutating the origina…How to Perform Welch's t-Test in Python
Calculate the Welch t-statistic and degrees of freedom for two samples with unequal variances using Python's statistics module.
import math
from statistics import mean, variance
def welch_t_test(sample1, sample2):
n1, n2 = len(sample1), len(sample2)
mean1, mean2 = mean(sample1), mean(sample2)
var1, var2 = variance(sample1), variance(sample2)
# Welch's t statistic
t_stat = (mean1 - mean2) / math.sqrt(var1 / n1 + var2 / n2…How to Run a Fisher Exact Test in Python
Compute the two-sided Fisher exact test p-value for a 2x2 contingency table using pure Python and the math module.
from math import comb, factorial
from itertools import combinations
def hypergeometric_probability(a, b, c, d):
"""Probability of observing table [[a, b], [c, d]] under the null."""
row1 = a + b
row2 = c + d
col1 = a + c
col2 = b + d
total = row1 + row2
return (comb(row1, a) * comb(row2, …How to Run a Permutation Test in Python
Run a Monte Carlo permutation test to compute a p-value for comparing two group means without parametric assumptions.
import random
import statistics
def permutation_test(group_a, group_b, n_permutations=10000, seed=42):
random.seed(seed)
combined = group_a + group_b
observed_diff = abs(statistics.mean(group_a) - statistics.mean(group_b))
count = 0
n = len(group_a)
for _ in range(n_permutations):
…How to Simulate Fixed-Horizon Testing in Python
Simulate a fixed-horizon experiment by labeling data before the horizon as warmup and after as active/inactive, then summarize via CSV.
import csv
import io
def fixed_horizon_mock(data: list[tuple[float, float, float]], horizon: int) -> str:
"""Simulate fixed-horizon testing, then summarize with CSV output."""
output = io.StringIO()
writer = csv.writer(output)
writer.writerow(["day", "value", "signal", "status"])
for day, value,…How to Simulate Geo Experiments in Python
Build a mock geo experiment simulator with ramp-up/down periods, measuring weekly lift between treatment and control markets.
import random
import math
from dataclasses import dataclass
@dataclass
class GeoMarket:
name: str
base_demand: float
geo_coefficient: float
def simulate_geo_experiment(markets, weeks=12, control_weeks=6):
"""
Simulates a geo experiment with ramp-up and ramp-down periods.
Returns weekly lift p…How to create a global control holdout group in Python
This code implements a deterministic global control holdout group, randomly selecting a fraction of users to be excluded from feature rollouts for experiment validation.
import random
class GlobalControl:
def __init__(self, population_size, holdout_fraction=0.2, seed=42):
random.seed(seed)
self.population_size = population_size
self.holdout_fraction = holdout_fraction
self.holdout_size = int(population_size * holdout_fraction)
self.holdout_…How to hash user IDs to experiment buckets in Python
Deterministically map a user ID to an experiment bucket using MD5 hashing, ensuring stable and consistent assignment for A/B testing.
import hashlib
def hash_user_to_bucket(user_id: str, num_buckets: int = 10) -> int:
"""Deterministically map a user ID to an experiment bucket (0..num_buckets-1)."""
digest = hashlib.md5(user_id.encode("utf-8")).hexdigest()
return int(digest, 16) % num_buckets
if __name__ == "__main__":
mock_users …How to join assignment logs with outcomes in Python
Merge submission log entries with grading outcomes using left join and full outer join patterns in pure Python.
from datetime import datetime, timedelta
class AssignmentLog:
def __init__(self):
self.logs = [
{"assignment_id": 101, "student_id": "S001", "submitted_at": "2024-03-01 10:30:00"},
{"assignment_id": 101, "student_id": "S002", "submitted_at": "2024-03-02 14:15:00"},
{"as…How to simulate a contextual bandit in Python
Simulate a contextual multi-armed bandit with random features and epsilon-greedy action selection in Python.
import random
class ContextualBandit:
def __init__(self, n_actions=3, n_features=4):
self.n_actions = n_actions
self.n_features = n_features
self.theta = [random.random() for _ in range(n_actions * n_features)]
def mock_context(self):
return [random.uniform(-1, 1) for _ in ra…Simulate a Ramp Rollout Percentage in Python
Simulates a percentage-based ramp rollout with deterministic seeding, returning success/failure/in-progress counts for a mock user population.
import random
from enum import Enum
class RolloutStatus(Enum):
SUCCESS = "success"
FAILED = "failed"
IN_PROGRESS = "in_progress"
def simulate_ramp_rollout(total_users: int, percentage: int, seed: int = 42) -> dict:
"""
Simulates a mock ramp rollout for a given percentage of users.
Returns sta…Synthetic Control in Python: Mock Example
Implements synthetic control from scratch: learns donor weights via ridge regression on pre-period data, then predicts a counterfactual for the treated unit.
import numpy as np
class SyntheticControl:
def __init__(self, data, treated_index, pre_periods, post_periods):
self.data = np.array(data, dtype=float)
self.treated_index = treated_index
self.pre_periods = pre_periods
self.post_periods = post_periods
def fit_weights(sel…Thompson Sampling Mock Bandit in Python
Implement a Thompson sampling multi-armed bandit to explore and exploit reward probabilities across multiple options, updating Beta distributions over time.
import random
class ThompsonSamplingBandit:
def __init__(self, num_arms, alpha=1.0, beta=1.0):
self.num_arms = num_arms
self.alpha = [alpha] * num_arms
self.beta = [beta] * num_arms
def select_arm(self):
samples = [random.betavariate(a, b) for a, b in zip(self.alpha, self.beta…Browse by section
Each section groups closely related Python snippets.
A/B testing & experimentation — Python code examples
What you will find here
This page collects a/b testing & experimentation snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.