A/B testing & experimentation
User bucketing, experiment metrics, statistical comparison, and rollout guardrails.
Generate a Mock Multi-Armed Bandit Report in Python
Simulate a multi-armed bandit experiment with random pulls and rewards, then output a JSON report with per-arm statistics.
import random
import json
def generate_mock_bandit_report(num_arms=5, num_rounds=100, seed=42):
random.seed(seed)
arms = ["A", "B", "C", "D", "E"][:num_arms]
true_means = {arm: random.uniform(0.3, 0.7) for arm in arms}
pulls = {arm: 0 for arm in arms}
rewards = {arm: 0 for arm in arms}
for _ …How to Build a Simple Binary Protocol Parser Mock in Python
Defines a mock binary protocol with field definitions, encoding, and decoding to simulate network packet parsing for A/B testing and experiment setup.
class SimpleProtocol:
def __init__(self, name, version):
self.name = name
self.version = version
self.fields = []
def add_field(self, field_name, field_size):
self.fields.append((field_name, field_size))
def parse(self, data):
if len(data) != sum(size for _, size i…How to Compute CUPED Variance Reduction in Python
Implement CUPED in Python to reduce variance of A/B test treatment effect estimates using pre-experiment covariates.
import numpy as np
def compute_cuped_reduction(control, variant, covariate):
"""
Compute variance reduction using CUPED (Controlled Experiment with
Pre-Experiment Data). Uses pre-experiment covariate values to
reduce variance of the treatment effect estimate.
"""
control = np.asarray(control, …How to Define a Mock Primary Metric in Python
Define a mock primary metric object with a name, value, and unit, and serialize it to a dictionary for experimentation and testing.
class Metric:
def __init__(self, name, value, unit=None):
self.name = name
self.value = value
self.unit = unit
def to_dict(self):
result = {"name": self.name, "value": self.value}
if self.unit:
result["unit"] = self.unit
return result
def __repr…How to Evaluate Feature Flags in Python
A Python function that evaluates boolean feature flags with user-specific overrides, returning whether a flag is enabled and the reason for the decision.
import json
def evaluate_feature_flag(feature_name, context, flag_configs):
"""
Evaluates a boolean feature flag given a context dictionary.
Args:
feature_name: The name of the feature flag.
context: A dictionary of user/request context (e.g., {"user_id": "123"}).
flag_configs: A …How to Generate Multivariate JSON Mock Data in Python
This script generates mock multivariate JSON-compatible data with measurements and boolean flags for testing and experimentation pipelines.
import json
def multivariate_mock(row_count: int = 3) -> list:
"""Generate mock multivariate data as list of JSON-compatible dicts."""
records = []
for i in range(row_count):
record = {
"id": i + 1,
"measurements": {
"temperature": 20.5 + i * 1.5,
…How to Generate an Orthogonal Array for A/B Testing in Python
Generate a mock orthogonal array for multi-layer experiments with NumPy, ensuring balanced level combinations across experiment groups.
import numpy as np
def orthogonal_mock_layers(n_experiments: int, n_layers: int, n_levels: int) -> np.ndarray:
"""Generate an orthogonal array for multi-layer experiment design using base-level logic."""
ortho = np.indices((n_levels,) * n_layers).reshape(n_layers, -1).T
ortho = ortho % n_levels # Classic…How to Hash a User ID to an Experiment Bucket in Python
Deterministically map a user ID to one of N experiment buckets using MD5 hashing and modulo arithmetic.
import hashlib
def hash_to_bucket(user_id: str, num_buckets: int = 10) -> int:
"""Deterministically map a user_id to a bucket (0 to num_buckets-1)."""
digest = hashlib.md5(user_id.encode("utf-8")).hexdigest()
return int(digest[:8], 16) % num_buckets
if __name__ == "__main__":
# Mock experiment: split…How to Mock Mutual Exclusion for A/B Experiment Groups in Python
Simulate mutual exclusion for experiment groups using a thread-safe lock, ensuring only one member updates the shared counter at a time.
import threading
import time
import random
class CountingGate:
"""A mock mutual exclusion gate using a lock."""
def __init__(self):
self.counter = 0
self.lock = threading.Lock()
def enter(self, group_id, member_id):
with self.lock:
current = self.counter
t…How to Mock Stratified Assignment by Segment in Python
Simulate stratified assignment for A/B experiments by sampling a fixed proportion of units from each segment, with deterministic seeds for reproducibility.
import random
def stratified_assignment(segments, seed=None):
"""
Mock stratified assignment: given a dict of segment -> population size,
return a dict of segment -> sampled unit ids (deterministic with seed).
"""
if seed is not None:
random.seed(seed)
rng = random.Random(seed)
res…How to Mock a Remote Config Fetch in Python
Simulate a remote config API response with metadata, timestamps, and mock data for testing or local development.
import json
from datetime import datetime
from typing import Any, Dict
def fetch_remote_config(mock_data: Dict[str, Any]) -> Dict[str, Any]:
"""Simulate fetching a remote config with metadata and timestamps."""
return {
"status": "success",
"source": "mock",
"fetched_at": datetime.utcn…How to Mock an Exposure Event Log Record in Python
Generate a realistic exposure event record with UUID, UTC timestamp, and risk level for testing or experimentation.
import uuid
from datetime import datetime, timezone
def mock_exposure_event(person_id: str, location: str, duration_minutes: int) -> dict:
return {
"event_id": str(uuid.uuid4()),
"person_id": person_id,
"location": location,
"duration_minutes": duration_minutes,
"timestamp…How to Perform Intent-to-Treat Analysis in Python
Runs an intent-to-treat analysis on mock A/B test data, comparing outcomes by initial group assignment with a t-test for significance.
import pandas as pd
import numpy as np
def intent_to_treat_analysis(data):
"""Perform intent-to-treat (ITT) analysis.
ITT compares outcomes based on initial treatment assignment,
regardless of whether participants actually received the treatment.
"""
# Create a copy to avoid mutating the origina…How to Simulate Fixed-Horizon Testing in Python
Simulate a fixed-horizon experiment by labeling data before the horizon as warmup and after as active/inactive, then summarize via CSV.
import csv
import io
def fixed_horizon_mock(data: list[tuple[float, float, float]], horizon: int) -> str:
"""Simulate fixed-horizon testing, then summarize with CSV output."""
output = io.StringIO()
writer = csv.writer(output)
writer.writerow(["day", "value", "signal", "status"])
for day, value,…How to Simulate Geo Experiments in Python
Build a mock geo experiment simulator with ramp-up/down periods, measuring weekly lift between treatment and control markets.
import random
import math
from dataclasses import dataclass
@dataclass
class GeoMarket:
name: str
base_demand: float
geo_coefficient: float
def simulate_geo_experiment(markets, weeks=12, control_weeks=6):
"""
Simulates a geo experiment with ramp-up and ramp-down periods.
Returns weekly lift p…How to create a global control holdout group in Python
This code implements a deterministic global control holdout group, randomly selecting a fraction of users to be excluded from feature rollouts for experiment validation.
import random
class GlobalControl:
def __init__(self, population_size, holdout_fraction=0.2, seed=42):
random.seed(seed)
self.population_size = population_size
self.holdout_fraction = holdout_fraction
self.holdout_size = int(population_size * holdout_fraction)
self.holdout_…How to hash user IDs to experiment buckets in Python
Deterministically map a user ID to an experiment bucket using MD5 hashing, ensuring stable and consistent assignment for A/B testing.
import hashlib
def hash_user_to_bucket(user_id: str, num_buckets: int = 10) -> int:
"""Deterministically map a user ID to an experiment bucket (0..num_buckets-1)."""
digest = hashlib.md5(user_id.encode("utf-8")).hexdigest()
return int(digest, 16) % num_buckets
if __name__ == "__main__":
mock_users …Browse by section
Each section groups closely related Python snippets.
A/B testing & experimentation — Python code examples
What you will find here
This page collects a/b testing & experimentation snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.