Reference library

A/B testing & experimentation

User bucketing, experiment metrics, statistical comparison, and rollout guardrails.

3 matches

Bonferroni Correction in Python

Applies the Bonferroni correction to a list of p-values to control the family-wise error rate when performing multiple comparisons.

statistics p-values multiple-comparisons

Python

import numpy as np

def bonferroni_correction(p_values, alpha=0.05):
    """Apply Bonferroni correction to a list of p-values."""
    n = len(p_values)
    corrected_alpha = alpha / n
    significant = [p < corrected_alpha for p in p_values]
    return corrected_alpha, significant

if __name__ == "__main__":
    # Moc…

16 0 Open

A/B testing & experimentation easy

How to Build a Guardrail Metrics Monitor in Python

This code implements a mock monitor that records metric values, checks them against thresholds, and summarizes pass/alert statistics.

metrics monitoring ab-testing

Python

import random
import time
from collections import defaultdict


class GuardrailMetricsMonitor:
    def __init__(self):
        self.metrics = defaultdict(list)
        self.thresholds = {
            "prompt_toxicity": 0.8,
            "response_length": 500,
            "latency_ms": 1000,
        }

    def record(s…

15 0 Open

A/B testing & experimentation easy

How to Create a Mock That Returns Inverse Counter Values in Python

Builds a Mock whose side_effect returns the inverse (1/count) of each Counter value, defaulting to 0.0 for unseen keys.

mock counter testing

Python

from collections import Counter
from unittest.mock import Mock

def inverse_mock(counter: Counter) -> Mock:
    """
    Return a Mock that mimics the inverse of a Counter:
    each key returns a value representing the inverse of its count.
    The Mock's side_effect maps keys to their inverse counts.
    """
    mock …

13 0 Open

Browse by section

Each section groups closely related Python snippets.