A/B testing & experimentation
User bucketing, experiment metrics, statistical comparison, and rollout guardrails.
Benjamini Hochberg FDR Correction in Python
Implement the Benjamini-HHochberg false discovery rate (FDR) procedure in Python to control the expected proportion of false positives among rejected hypotheses.
import numpy as np
def benjamini_hochberg(p_values, alpha=0.05):
p_values = np.array(p_values)
n = len(p_values)
sorted_idx = np.argsort(p_values)
sorted_p = p_values[sorted_idx]
thresholds = (np.arange(1, n + 1) / n) * alpha
significant = sorted_p <= thresholds
if not significan…Check Covariate Balance in Python
Compute standardized mean differences and KS tests to check covariate balance between treatment and control groups in Python.
import numpy as np
from scipy import stats
def balance_check(treatment, covariate):
"""Check covariate balance between treatment and control groups."""
treat_vals = covariate[treatment == 1]
control_vals = covariate[treatment == 0]
# Standardized mean difference
pooled_std = np.sqrt((np.var(t…How to Compute Mann-Whitney U Test in Python
Compute the Mann-Whitney U statistic and p-value manually in Python with tie correction and a normal approximation for independent samples.
import numpy as np
from scipy import stats
def mann_whitney_u_mock(sample_a, sample_b):
"""Compute Mann-Whitney U and p-value manually."""
# Combine and rank
combined = sample_a + sample_b
n_a, n_b = len(sample_a), len(sample_b)
n_total = n_a + n_b
# Rank with ties handling (average ranks…How to Conduct a Two-Sample T-Test in Python
Performs Welch's t-test for two independent samples, computing the t-statistic, degrees of freedom, and p-value using NumPy and SciPy.
import numpy as np
def two_sample_t_test(sample1, sample2):
"""Perform Welch's t-test for two independent samples."""
n1, n2 = len(sample1), len(sample2)
mean1, mean2 = np.mean(sample1), np.mean(sample2)
var1, var2 = np.var(sample1, ddof=1), np.var(sample2, ddof=1)
# Standard error of difference
…How to Perform Intent-to-Treat Analysis in Python
Runs an intent-to-treat analysis on mock A/B test data, comparing outcomes by initial group assignment with a t-test for significance.
import pandas as pd
import numpy as np
def intent_to_treat_analysis(data):
"""Perform intent-to-treat (ITT) analysis.
ITT compares outcomes based on initial treatment assignment,
regardless of whether participants actually received the treatment.
"""
# Create a copy to avoid mutating the origina…Thompson Sampling Mock Bandit in Python
Implement a Thompson sampling multi-armed bandit to explore and exploit reward probabilities across multiple options, updating Beta distributions over time.
import random
class ThompsonSamplingBandit:
def __init__(self, num_arms, alpha=1.0, beta=1.0):
self.num_arms = num_arms
self.alpha = [alpha] * num_arms
self.beta = [beta] * num_arms
def select_arm(self):
samples = [random.betavariate(a, b) for a, b in zip(self.alpha, self.beta…Browse by section
Each section groups closely related Python snippets.
A/B testing & experimentation — Python code examples
What you will find here
This page collects a/b testing & experimentation snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.