How to Generate an Orthogonal Array for A/B Testing in Python

Generate a mock orthogonal array for multi-layer experiments with NumPy, ensuring balanced level combinations across experiment groups.

Medium Python 3.9+ Aug 9, 2026 A/B testing & experimentation 14 views 0 copies

ab-testing orthogonal-array numpy experiment-design balancing
Requires third-party packages — install first
pip install numpy

Python code

16 lines
Python 3.9+ 
import numpy as np

def orthogonal_mock_layers(n_experiments: int, n_layers: int, n_levels: int) -> np.ndarray:
    """Generate an orthogonal array for multi-layer experiment design using base-level logic."""
    ortho = np.indices((n_levels,) * n_layers).reshape(n_layers, -1).T
    ortho = ortho % n_levels  # Classic orthogonal tagging per layer
    return np.tile(ortho, (n_experiments // len(ortho) + 1, 1))[:n_experiments]

if __name__ == "__main__":
    experiments = orthogonal_mock_layers(n_experiments=10, n_layers=3, n_levels=2)
    print("Layer/level assignments (rows=experiments, cols=layers):")
    print(experiments)
    print("\nOrthogonality check: unique layer-pair combinations")
    for a, b in [(0,1), (0,2), (1,2)]:
        combos = set(map(tuple, experiments[:, [a, b]]))
        print(f"Layer {a+1}-{b+1}: {len(combos)} combinations")

Output

stdout 
Layer/level assignments (rows=experiments, cols=layers):
[[0 0 0]
 [1 0 0]
 [0 1 0]
 [1 1 0]
 [0 0 1]
 [1 0 1]
 [0 1 1]
 [1 1 1]
 [0 0 0]
 [1 0 0]]

Orthogonality check: unique layer-pair combinations
Layer 1-2: 4 combinations
Layer 1-3: 4 combinations
Layer 2-3: 4 combinations

How it works

The function uses np.indices to create a grid of all possible level combinations across layers, then reshapes it to a 2D array where each row represents a distinct experiment. The modulo operation (% n_levels) enforces orthogonality by cycling levels, though for base-level combinations it's redundant. np.tile repeats the base array enough times to cover the requested number of experiments, and slicing truncates to exactly n_experiments. This creates a mock orthogonal array that ensures each layer-pair combination appears the same number of times, which is useful for balanced multi-factor test designs.

Common mistakes

  • Forgetting to install NumPy (`pip install numpy`) and getting ImportError
  • Assuming the output is a true orthogonal array—this is a mock—always verify pair coverage
  • Not handling cases where `n_experiments` is not a multiple of the base array length, leading to repeated patterns that may bias results

Variations

  1. Use `itertools.product` with `list(range(n_levels))` repeated `n_layers` times for a pure-Python version
  2. For production-grade arrays, consider `pyDOE2` or `SALib` which generate actual orthogonal arrays with guaranteed properties

Real-world use cases

  • Mocking experiment assignments in unit tests before building the real randomization service.
  • Simulating multi-factor A/B tests (e.g., UI changes, algorithm variants) to sanity-check analysis code without live traffic.
  • Generating a balanced training/eval split matrix for machine learning experiments where multiple config layers are tested.

Sponsored

Run locally

This sample needs third-party packages, so it cannot run in the browser IDE. Copy the code above, install the packages shown at the top, then run it in your own Python environment.

More from A/B testing & experimentation

Keep learning

Related tutorials and quizzes for this topic.

Tutorials

Quizzes