A/B testing & experimentation
User bucketing, experiment metrics, statistical comparison, and rollout guardrails.
How to Perform Intent-to-Treat Analysis in Python
Runs an intent-to-treat analysis on mock A/B test data, comparing outcomes by initial group assignment with a t-test for significance.
import pandas as pd
import numpy as np
def intent_to_treat_analysis(data):
"""Perform intent-to-treat (ITT) analysis.
ITT compares outcomes based on initial treatment assignment,
regardless of whether participants actually received the treatment.
"""
# Create a copy to avoid mutating the origina…How to Perform Welch's t-Test in Python
Calculate the Welch t-statistic and degrees of freedom for two samples with unequal variances using Python's statistics module.
import math
from statistics import mean, variance
def welch_t_test(sample1, sample2):
n1, n2 = len(sample1), len(sample2)
mean1, mean2 = mean(sample1), mean(sample2)
var1, var2 = variance(sample1), variance(sample2)
# Welch's t statistic
t_stat = (mean1 - mean2) / math.sqrt(var1 / n1 + var2 / n2…How to Run a Fisher Exact Test in Python
Compute the two-sided Fisher exact test p-value for a 2x2 contingency table using pure Python and the math module.
from math import comb, factorial
from itertools import combinations
def hypergeometric_probability(a, b, c, d):
"""Probability of observing table [[a, b], [c, d]] under the null."""
row1 = a + b
row2 = c + d
col1 = a + c
col2 = b + d
total = row1 + row2
return (comb(row1, a) * comb(row2, …How to Run a Permutation Test in Python
Run a Monte Carlo permutation test to compute a p-value for comparing two group means without parametric assumptions.
import random
import statistics
def permutation_test(group_a, group_b, n_permutations=10000, seed=42):
random.seed(seed)
combined = group_a + group_b
observed_diff = abs(statistics.mean(group_a) - statistics.mean(group_b))
count = 0
n = len(group_a)
for _ in range(n_permutations):
…How to Simulate Fixed-Horizon Testing in Python
Simulate a fixed-horizon experiment by labeling data before the horizon as warmup and after as active/inactive, then summarize via CSV.
import csv
import io
def fixed_horizon_mock(data: list[tuple[float, float, float]], horizon: int) -> str:
"""Simulate fixed-horizon testing, then summarize with CSV output."""
output = io.StringIO()
writer = csv.writer(output)
writer.writerow(["day", "value", "signal", "status"])
for day, value,…How to Simulate Geo Experiments in Python
Build a mock geo experiment simulator with ramp-up/down periods, measuring weekly lift between treatment and control markets.
import random
import math
from dataclasses import dataclass
@dataclass
class GeoMarket:
name: str
base_demand: float
geo_coefficient: float
def simulate_geo_experiment(markets, weeks=12, control_weeks=6):
"""
Simulates a geo experiment with ramp-up and ramp-down periods.
Returns weekly lift p…How to create a global control holdout group in Python
This code implements a deterministic global control holdout group, randomly selecting a fraction of users to be excluded from feature rollouts for experiment validation.
import random
class GlobalControl:
def __init__(self, population_size, holdout_fraction=0.2, seed=42):
random.seed(seed)
self.population_size = population_size
self.holdout_fraction = holdout_fraction
self.holdout_size = int(population_size * holdout_fraction)
self.holdout_…How to hash user IDs to experiment buckets in Python
Deterministically map a user ID to an experiment bucket using MD5 hashing, ensuring stable and consistent assignment for A/B testing.
import hashlib
def hash_user_to_bucket(user_id: str, num_buckets: int = 10) -> int:
"""Deterministically map a user ID to an experiment bucket (0..num_buckets-1)."""
digest = hashlib.md5(user_id.encode("utf-8")).hexdigest()
return int(digest, 16) % num_buckets
if __name__ == "__main__":
mock_users …How to join assignment logs with outcomes in Python
Merge submission log entries with grading outcomes using left join and full outer join patterns in pure Python.
from datetime import datetime, timedelta
class AssignmentLog:
def __init__(self):
self.logs = [
{"assignment_id": 101, "student_id": "S001", "submitted_at": "2024-03-01 10:30:00"},
{"assignment_id": 101, "student_id": "S002", "submitted_at": "2024-03-02 14:15:00"},
{"as…How to simulate a contextual bandit in Python
Simulate a contextual multi-armed bandit with random features and epsilon-greedy action selection in Python.
import random
class ContextualBandit:
def __init__(self, n_actions=3, n_features=4):
self.n_actions = n_actions
self.n_features = n_features
self.theta = [random.random() for _ in range(n_actions * n_features)]
def mock_context(self):
return [random.uniform(-1, 1) for _ in ra…Browse by section
Each section groups closely related Python snippets.
A/B testing & experimentation — Python code examples
What you will find here
This page collects a/b testing & experimentation snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.