A/B testing & experimentation
User bucketing, experiment metrics, statistical comparison, and rollout guardrails.
Bonferroni Correction in Python
Applies the Bonferroni correction to a list of p-values to control the family-wise error rate when performing multiple comparisons.
import numpy as np
def bonferroni_correction(p_values, alpha=0.05):
"""Apply Bonferroni correction to a list of p-values."""
n = len(p_values)
corrected_alpha = alpha / n
significant = [p < corrected_alpha for p in p_values]
return corrected_alpha, significant
if __name__ == "__main__":
# Moc…Generate a Mock Multi-Armed Bandit Report in Python
Simulate a multi-armed bandit experiment with random pulls and rewards, then output a JSON report with per-arm statistics.
import random
import json
def generate_mock_bandit_report(num_arms=5, num_rounds=100, seed=42):
random.seed(seed)
arms = ["A", "B", "C", "D", "E"][:num_arms]
true_means = {arm: random.uniform(0.3, 0.7) for arm in arms}
pulls = {arm: 0 for arm in arms}
rewards = {arm: 0 for arm in arms}
for _ …How to Build a Guardrail Metrics Monitor in Python
This code implements a mock monitor that records metric values, checks them against thresholds, and summarizes pass/alert statistics.
import random
import time
from collections import defaultdict
class GuardrailMetricsMonitor:
def __init__(self):
self.metrics = defaultdict(list)
self.thresholds = {
"prompt_toxicity": 0.8,
"response_length": 500,
"latency_ms": 1000,
}
def record(s…How to Calculate Minimum Sample Size for a T-Test in Python
Compute the minimum sample size per group for a two-sample t-test using effect size, significance level, and statistical power.
import math
from scipy.stats import norm
def min_sample_size(effect_size, alpha=0.05, power=0.8):
"""
Calculate minimum sample size for a two-sample t-test (equal groups).
Args:
effect_size: Cohen's d (standardized mean difference)
alpha: significance level (Type I error)
power: …How to Calculate Secondary Metrics in Python
Computes distribution, variability, and spread of a numeric dataset using Python's statistics and collections modules.
import random
import statistics
from collections import Counter
def explore_secondary_metrics(data):
"""Calculate secondary metrics: distribution, variability, and spread."""
if not data:
return "No data provided"
total = sum(data)
mean = statistics.mean(data)
median = statistics.medi…How to Mock a Confidence Interval for a Proportion in Python
Simulate a Bernoulli sample and compute a 95% confidence interval for a proportion using the normal approximation in Python.
import random
import math
def mock_ci(n=100, p_true=0.5, z=1.96, seed=42):
"""Simulate a sample proportion and compute its 95% confidence interval."""
random.seed(seed)
successes = sum(1 for _ in range(n) if random.random() < p_true)
p_hat = successes / n
se = math.sqrt(p_hat * (1 - p_hat) / n)
…Browse by section
Each section groups closely related Python snippets.
A/B testing & experimentation — Python code examples
What you will find here
This page collects a/b testing & experimentation snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.