Reference library

A/B testing & experimentation

User bucketing, experiment metrics, statistical comparison, and rollout guardrails.

4 matches

How to Build a Guardrail Metrics Monitor in Python

This code implements a mock monitor that records metric values, checks them against thresholds, and summarizes pass/alert statistics.

metrics monitoring ab-testing

Python

import random
import time
from collections import defaultdict


class GuardrailMetricsMonitor:
    def __init__(self):
        self.metrics = defaultdict(list)
        self.thresholds = {
            "prompt_toxicity": 0.8,
            "response_length": 500,
            "latency_ms": 1000,
        }

    def record(s…

15 0 Open

A/B testing & experimentation easy

How to Build a Simple Binary Protocol Parser Mock in Python

Defines a mock binary protocol with field definitions, encoding, and decoding to simulate network packet parsing for A/B testing and experiment setup.

binary protocol mock

Python

class SimpleProtocol:
    def __init__(self, name, version):
        self.name = name
        self.version = version
        self.fields = []

    def add_field(self, field_name, field_size):
        self.fields.append((field_name, field_size))

    def parse(self, data):
        if len(data) != sum(size for _, size i…

12 0 Open

A/B testing & experimentation easy

How to Create a Mock That Returns Inverse Counter Values in Python

Builds a Mock whose side_effect returns the inverse (1/count) of each Counter value, defaulting to 0.0 for unseen keys.

mock counter testing

Python

from collections import Counter
from unittest.mock import Mock

def inverse_mock(counter: Counter) -> Mock:
    """
    Return a Mock that mimics the inverse of a Counter:
    each key returns a value representing the inverse of its count.
    The Mock's side_effect maps keys to their inverse counts.
    """
    mock …

13 0 Open

A/B testing & experimentation medium

How to Simulate Geo Experiments in Python

Build a mock geo experiment simulator with ramp-up/down periods, measuring weekly lift between treatment and control markets.

geo-experiment ab-testing simulation

Python

import random
import math
from dataclasses import dataclass

@dataclass
class GeoMarket:
    name: str
    base_demand: float
    geo_coefficient: float

def simulate_geo_experiment(markets, weeks=12, control_weeks=6):
    """
    Simulates a geo experiment with ramp-up and ramp-down periods.
    Returns weekly lift p…

18 0 Open

Browse by section

Each section groups closely related Python snippets.