A/B testing & experimentation
User bucketing, experiment metrics, statistical comparison, and rollout guardrails.
How to Simulate Fixed-Horizon Testing in Python
Simulate a fixed-horizon experiment by labeling data before the horizon as warmup and after as active/inactive, then summarize via CSV.
import csv
import io
def fixed_horizon_mock(data: list[tuple[float, float, float]], horizon: int) -> str:
"""Simulate fixed-horizon testing, then summarize with CSV output."""
output = io.StringIO()
writer = csv.writer(output)
writer.writerow(["day", "value", "signal", "status"])
for day, value,…How to Simulate Geo Experiments in Python
Build a mock geo experiment simulator with ramp-up/down periods, measuring weekly lift between treatment and control markets.
import random
import math
from dataclasses import dataclass
@dataclass
class GeoMarket:
name: str
base_demand: float
geo_coefficient: float
def simulate_geo_experiment(markets, weeks=12, control_weeks=6):
"""
Simulates a geo experiment with ramp-up and ramp-down periods.
Returns weekly lift p…How to create a global control holdout group in Python
This code implements a deterministic global control holdout group, randomly selecting a fraction of users to be excluded from feature rollouts for experiment validation.
import random
class GlobalControl:
def __init__(self, population_size, holdout_fraction=0.2, seed=42):
random.seed(seed)
self.population_size = population_size
self.holdout_fraction = holdout_fraction
self.holdout_size = int(population_size * holdout_fraction)
self.holdout_…How to hash user IDs to experiment buckets in Python
Deterministically map a user ID to an experiment bucket using MD5 hashing, ensuring stable and consistent assignment for A/B testing.
import hashlib
def hash_user_to_bucket(user_id: str, num_buckets: int = 10) -> int:
"""Deterministically map a user ID to an experiment bucket (0..num_buckets-1)."""
digest = hashlib.md5(user_id.encode("utf-8")).hexdigest()
return int(digest, 16) % num_buckets
if __name__ == "__main__":
mock_users …How to join assignment logs with outcomes in Python
Merge submission log entries with grading outcomes using left join and full outer join patterns in pure Python.
from datetime import datetime, timedelta
class AssignmentLog:
def __init__(self):
self.logs = [
{"assignment_id": 101, "student_id": "S001", "submitted_at": "2024-03-01 10:30:00"},
{"assignment_id": 101, "student_id": "S002", "submitted_at": "2024-03-02 14:15:00"},
{"as…Simulate a Ramp Rollout Percentage in Python
Simulates a percentage-based ramp rollout with deterministic seeding, returning success/failure/in-progress counts for a mock user population.
import random
from enum import Enum
class RolloutStatus(Enum):
SUCCESS = "success"
FAILED = "failed"
IN_PROGRESS = "in_progress"
def simulate_ramp_rollout(total_users: int, percentage: int, seed: int = 42) -> dict:
"""
Simulates a mock ramp rollout for a given percentage of users.
Returns sta…Thompson Sampling Mock Bandit in Python
Implement a Thompson sampling multi-armed bandit to explore and exploit reward probabilities across multiple options, updating Beta distributions over time.
import random
class ThompsonSamplingBandit:
def __init__(self, num_arms, alpha=1.0, beta=1.0):
self.num_arms = num_arms
self.alpha = [alpha] * num_arms
self.beta = [beta] * num_arms
def select_arm(self):
samples = [random.betavariate(a, b) for a, b in zip(self.alpha, self.beta…UCB1 Bandit Algorithm in Python
This code implements the UCB1 multi-armed bandit algorithm, balancing exploration and exploitation to identify the best arm while maximizing cumulative reward.
import math
import random
def ucb1(means, n_iterations=1000, exploration_weight=2.0):
"""Run UCB1 bandit algorithm on arms with given true means."""
n_arms = len(means)
counts = [0] * n_arms
rewards = [0.0] * n_arms
for t in range(1, n_iterations + 1):
# UCB1 selection
if t <…Browse by section
Each section groups closely related Python snippets.
A/B testing & experimentation — Python code examples
What you will find here
This page collects a/b testing & experimentation snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.