Reference library

A/B testing & experimentation

User bucketing, experiment metrics, statistical comparison, and rollout guardrails.

3 matches

How to Build a Guardrail Metrics Monitor in Python

This code implements a mock monitor that records metric values, checks them against thresholds, and summarizes pass/alert statistics.

metrics monitoring ab-testing

Python

import random
import time
from collections import defaultdict


class GuardrailMetricsMonitor:
    def __init__(self):
        self.metrics = defaultdict(list)
        self.thresholds = {
            "prompt_toxicity": 0.8,
            "response_length": 500,
            "latency_ms": 1000,
        }

    def record(s…

15 0 Open

A/B testing & experimentation medium

How to Run a Permutation Test in Python

Run a Monte Carlo permutation test to compute a p-value for comparing two group means without parametric assumptions.

permutation-test statistics ab-testing

Python

import random
import statistics

def permutation_test(group_a, group_b, n_permutations=10000, seed=42):
    random.seed(seed)
    combined = group_a + group_b
    observed_diff = abs(statistics.mean(group_a) - statistics.mean(group_b))
    
    count = 0
    n = len(group_a)
    for _ in range(n_permutations):
       …

15 0 Open

A/B testing & experimentation easy

How to Simulate Fixed-Horizon Testing in Python

Simulate a fixed-horizon experiment by labeling data before the horizon as warmup and after as active/inactive, then summarize via CSV.

ab-testing simulation csv

Python

import csv
import io


def fixed_horizon_mock(data: list[tuple[float, float, float]], horizon: int) -> str:
    """Simulate fixed-horizon testing, then summarize with CSV output."""
    output = io.StringIO()
    writer = csv.writer(output)
    writer.writerow(["day", "value", "signal", "status"])

    for day, value,…

13 0 Open

Browse by section

Each section groups closely related Python snippets.