Reference library

A/B testing & experimentation

User bucketing, experiment metrics, statistical comparison, and rollout guardrails.

5 matches

Check Covariate Balance in Python

Compute standardized mean differences and KS tests to check covariate balance between treatment and control groups in Python.

covariate balance ab-testing

Python

import numpy as np
from scipy import stats

def balance_check(treatment, covariate):
    """Check covariate balance between treatment and control groups."""
    treat_vals = covariate[treatment == 1]
    control_vals = covariate[treatment == 0]
    
    # Standardized mean difference
    pooled_std = np.sqrt((np.var(t…

13 0 Open

A/B testing & experimentation medium

Check Sample Ratio Mismatch in Python

Estimates the probability that a simple random sample's proportion differs from the population proportion by more than 10% using simulation.

simulation statistics ab-testing

Python

import random


def sample_ratio_mismatch(population_size: int, sample_size: int, p: float) -> float:
    """
    Estimate the probability that a simple random sample's proportion
    differs from the population proportion by more than 10%.
    """
    total_counts = [0, 0]
    for _ in range(10000):
        sample = …

15 0 Open

A/B testing & experimentation medium

Delta Method for Ratio Metrics in A/B Testing with Python

Computes the confidence interval for the difference between two ratio metrics using the delta method, with mock A/B test data.

delta-method ab-testing ratio-metrics

Python

import numpy as np
from scipy.stats import norm


def delta_method_ratio_delta(control: np.ndarray, treatment: np.ndarray, confidence: float = 0.95):
    """Estimate confidence interval for ratio metric using delta method.

    Args:
        control: numerator/denominator pairs from control group (n x 2 array)
       …

15 0 Open

A/B testing & experimentation easy

Difference in Differences Mock in Python

Generate mock panel data with a known treatment effect and compute a difference-in-differences estimate using group and period means.

did pandas simulation

Python

import numpy as np
import pandas as pd

# Generate mock panel data: 2 groups (control=0, treatment=1) × 2 periods (pre=0, post=1)
rng = np.random.default_rng(42)
n_per_cell = 50

data = []
for group in [0, 1]:
    for period in [0, 1]:
        # True effect: treatment increases outcome by 5 in the post period
        …

16 0 Open

A/B testing & experimentation medium

How to Mock Sequential Calls in Python with unittest.mock

Use Mock.side_effect to return a different result for each sequential call and verify the call order with assert_has_calls.

mock unittest testing

Python

import unittest
from unittest.mock import Mock

class Service:
    def fetch(self, item_id):
        raise NotImplementedError

def process_items(service, ids):
    results = []
    for item_id in ids:
        result = service.fetch(item_id)
        results.append(result)
    return results

if __name__ == "__main__":…

14 0 Open

Browse by section

Each section groups closely related Python snippets.