How to create a global control holdout group in Python

This code implements a deterministic global control holdout group, randomly selecting a fraction of users to be excluded from feature rollouts for experiment validation.

Easy Python 3.9+ Aug 9, 2026 A/B testing & experimentation 11 views 0 copies

ab-testing holdout global-control experimentation random

Python code

28 lines

Python 3.9+

import random

class GlobalControl:
    def __init__(self, population_size, holdout_fraction=0.2, seed=42):
        random.seed(seed)
        self.population_size = population_size
        self.holdout_fraction = holdout_fraction
        self.holdout_size = int(population_size * holdout_fraction)
        self.holdout_ids = set(random.sample(range(population_size), self.holdout_size))

    def is_holdout(self, entity_id):
        return entity_id in self.holdout_ids

    def get_holdout_ids(self):
        return sorted(self.holdout_ids)

    def get_treatment_ids(self):
        return [i for i in range(self.population_size) if i not in self.holdout_ids]

    def assign_global_control(self, entity_id):
        return "holdout" if self.is_holdout(entity_id) else "treatment"

if __name__ == "__main__":
    control = GlobalControl(population_size=100, holdout_fraction=0.2, seed=7)
    sample_ids = [5, 17, 42, 88, 99]
    for eid in sample_ids:
        print(f"ID {eid}: {control.assign_global_control(eid)}")
    print(f"Holdout count: {len(control.get_holdout_ids())}, Treatment count: {len(control.get_treatment_ids())}")

Output

stdout

ID 5: treatment
ID 17: holdout
ID 42: treatment
ID 88: treatment
ID 99: treatment
Holdout count: 20, Treatment count: 80

How it works

The random.seed ensures that the holdout assignment is reproducible across runs, which is critical for consistent experiment results. By pre-selecting the holdout IDs once at initialization, the class provides fast O(1) membership checks via a set. The assign_global_control method returns a clear label for each entity, making it easy to integrate into a feature flag system. This design separates the selection logic from the assignment logic, keeping the class flexible and testable.

Common mistakes

Using `random.sample` without a seed, causing different holdout groups each time the script runs
Forgetting to cast `population_size * holdout_fraction` to an integer, leading to float indexing errors
Storing holdout IDs in a list instead of a set, making membership checks O(n) and slow for large populations

Variations

Use `random.shuffle` on a list of user IDs and split into holdout/treatment slices for a similar effect
Persist the holdout list to a database or file so it remains stable across service restarts

Real-world use cases

A product team reserves 20% of users as a global control group to measure the long-term impact of all feature releases without contamination.
An online platform uses a holdout group to validate that new recommendation algorithms don't degrade overall engagement metrics.
A streaming service maintains a stable holdout segment to compare the cumulative effect of multiple UI changes over several sprints.

How to create a global control holdout group in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from A/B testing & experimentation

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from A/B testing & experimentation

Keep learning

Tutorials

Quizzes