A/B testing & experimentation
User bucketing, experiment metrics, statistical comparison, and rollout guardrails.
Bayesian A/B Test Credible Interval in Python
Simulates A/B test data and computes posterior credible intervals and the probability that variant B outperforms A using Bayesian Beta-Binomial inference.
import numpy as np
from scipy import stats
# Simulated A/B test data
n_A = 1000
n_B = 1000
conversions_A = 120
conversions_B = 140
# Prior: Beta(1, 1) uniform
alpha_prior, beta_prior = 1, 1
# Posterior parameters
alpha_A = alpha_prior + conversions_A
beta_A = beta_prior + n_A - conversions_A
alpha_B = alpha_prior +…Chi-Square Test in Python for Conversion Mock Data
Compute the chi-square statistic and approximate p-value for a mock A/B conversion test using the standard library.
import math
from collections import Counter
def chi_square_statistic(observed):
"""
Compute chi-square statistic for a mock conversion test.
observed: dict mapping outcomes to observed frequencies.
"""
observed = Counter(observed)
n = sum(observed.values())
expected = n / len(observed) if …Delta Method for Ratio Metrics in A/B Testing with Python
Computes the confidence interval for the difference between two ratio metrics using the delta method, with mock A/B test data.
import numpy as np
from scipy.stats import norm
def delta_method_ratio_delta(control: np.ndarray, treatment: np.ndarray, confidence: float = 0.95):
"""Estimate confidence interval for ratio metric using delta method.
Args:
control: numerator/denominator pairs from control group (n x 2 array)
…How to Create an Interrupted Time Series Mock in Python
Generate simulated interrupted time series data with a pre/post-intervention trend, level shift, and noise to test segmented regression models.
import numpy as np
# Mock interrupted time series data
np.random.seed(42)
n_pre = 50
n_post = 50
time = np.arange(0, n_pre + n_post)
# Pre-intervention: linear trend + noise
pre_trend = 0.05 * time[:n_pre] + np.random.normal(0, 0.5, n_pre)
# Post-intervention: new slope + level shift + noise
post_trend = 0.05 * tim…How to Perform Intent-to-Treat Analysis in Python
Runs an intent-to-treat analysis on mock A/B test data, comparing outcomes by initial group assignment with a t-test for significance.
import pandas as pd
import numpy as np
def intent_to_treat_analysis(data):
"""Perform intent-to-treat (ITT) analysis.
ITT compares outcomes based on initial treatment assignment,
regardless of whether participants actually received the treatment.
"""
# Create a copy to avoid mutating the origina…How to Perform Welch's t-Test in Python
Calculate the Welch t-statistic and degrees of freedom for two samples with unequal variances using Python's statistics module.
import math
from statistics import mean, variance
def welch_t_test(sample1, sample2):
n1, n2 = len(sample1), len(sample2)
mean1, mean2 = mean(sample1), mean(sample2)
var1, var2 = variance(sample1), variance(sample2)
# Welch's t statistic
t_stat = (mean1 - mean2) / math.sqrt(var1 / n1 + var2 / n2…How to join assignment logs with outcomes in Python
Merge submission log entries with grading outcomes using left join and full outer join patterns in pure Python.
from datetime import datetime, timedelta
class AssignmentLog:
def __init__(self):
self.logs = [
{"assignment_id": 101, "student_id": "S001", "submitted_at": "2024-03-01 10:30:00"},
{"assignment_id": 101, "student_id": "S002", "submitted_at": "2024-03-02 14:15:00"},
{"as…Synthetic Control in Python: Mock Example
Implements synthetic control from scratch: learns donor weights via ridge regression on pre-period data, then predicts a counterfactual for the treated unit.
import numpy as np
class SyntheticControl:
def __init__(self, data, treated_index, pre_periods, post_periods):
self.data = np.array(data, dtype=float)
self.treated_index = treated_index
self.pre_periods = pre_periods
self.post_periods = post_periods
def fit_weights(sel…Browse by section
Each section groups closely related Python snippets.
A/B testing & experimentation — Python code examples
What you will find here
This page collects a/b testing & experimentation snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.