Reference library

A/B testing & experimentation

User bucketing, experiment metrics, statistical comparison, and rollout guardrails.

5 matches

Bonferroni Correction in Python

Applies the Bonferroni correction to a list of p-values to control the family-wise error rate when performing multiple comparisons.

statistics p-values multiple-comparisons

Python

import numpy as np

def bonferroni_correction(p_values, alpha=0.05):
    """Apply Bonferroni correction to a list of p-values."""
    n = len(p_values)
    corrected_alpha = alpha / n
    significant = [p < corrected_alpha for p in p_values]
    return corrected_alpha, significant

if __name__ == "__main__":
    # Moc…

17 0 Open

A/B testing & experimentation easy

How to Build a Guardrail Metrics Monitor in Python

This code implements a mock monitor that records metric values, checks them against thresholds, and summarizes pass/alert statistics.

metrics monitoring ab-testing

Python

import random
import time
from collections import defaultdict


class GuardrailMetricsMonitor:
    def __init__(self):
        self.metrics = defaultdict(list)
        self.thresholds = {
            "prompt_toxicity": 0.8,
            "response_length": 500,
            "latency_ms": 1000,
        }

    def record(s…

15 0 Open

A/B testing & experimentation easy

How to Create a Mock That Returns Inverse Counter Values in Python

Builds a Mock whose side_effect returns the inverse (1/count) of each Counter value, defaulting to 0.0 for unseen keys.

mock counter testing

Python

from collections import Counter
from unittest.mock import Mock

def inverse_mock(counter: Counter) -> Mock:
    """
    Return a Mock that mimics the inverse of a Counter:
    each key returns a value representing the inverse of its count.
    The Mock's side_effect maps keys to their inverse counts.
    """
    mock …

13 0 Open

A/B testing & experimentation easy

How to Create a Sticky Consistent Mock with unittest.mock in Python

Shows how to use unittest.mock.patch.object to mock a method consistently across multiple calls, returning a sticky value every time.

unittest mock testing

Python

from unittest.mock import patch

class Database:
    def fetch(self, key):
        return f"real value for {key}"

def get_value(db, key):
    return db.fetch(key)

if __name__ == "__main__":
    db = Database()
    with patch.object(db, "fetch", return_value="sticky value") as mock_fetch:
        result1 = get_value(…

15 0 Open

A/B testing & experimentation easy

How to Define a Mock Primary Metric in Python

Define a mock primary metric object with a name, value, and unit, and serialize it to a dictionary for experimentation and testing.

metrics mock ab-testing

Python

class Metric:
    def __init__(self, name, value, unit=None):
        self.name = name
        self.value = value
        self.unit = unit

    def to_dict(self):
        result = {"name": self.name, "value": self.value}
        if self.unit:
            result["unit"] = self.unit
        return result

    def __repr…

15 0 Open

Browse by section

Each section groups closely related Python snippets.