How to Simulate Fixed-Horizon Testing in Python

Simulate a fixed-horizon experiment by labeling data before the horizon as warmup and after as active/inactive, then summarize via CSV.

Easy Python 3.9+ Aug 9, 2026 A/B testing & experimentation 13 views 0 copies

ab-testing simulation csv mock-data experiment

Python code

35 lines
Python 3.9+
Run it 
import csv
import io


def fixed_horizon_mock(data: list[tuple[float, float, float]], horizon: int) -> str:
    """Simulate fixed-horizon testing, then summarize with CSV output."""
    output = io.StringIO()
    writer = csv.writer(output)
    writer.writerow(["day", "value", "signal", "status"])

    for day, value, signal in data:
        if day < horizon:
            status = "warmup"
        else:
            status = "active" if value > signal else "inactive"
        writer.writerow([day, value, signal, status])

    summary = {
        "total_days": len(data),
        "active_days": sum(1 for day, value, signal in data if day >= horizon and value > signal),
        "inactive_days": sum(1 for day, value, signal in data if day >= horizon and value <= signal),
    }
    return output.getvalue() + f"Summary: {summary}"


if __name__ == "__main__":
    mock_data = [
        (1, 10.0, 5.0),
        (2, 3.0, 5.0),
        (3, 7.0, 5.0),
        (4, 2.0, 5.0),
        (5, 9.0, 5.0),
        (6, 4.0, 5.0),
    ]
    print(fixed_horizon_mock(mock_data, horizon=4))

Output

stdout 
day,value,signal,status
1,10.0,5.0,warmup
2,3.0,5.0,warmup
3,7.0,5.0,warmup
4,2.0,5.0,active
5,9.0,5.0,active
6,4.0,5.0,inactive
Summary: {'total_days': 6, 'active_days': 2, 'inactive_days': 1}

How it works

The fixed_horizon_mock function simulates a common A/B testing pattern where data collected before a predefined horizon is considered pre-experiment (warmup) and excluded from decision metrics. Days after the horizon are classified based on whether the observed value exceeds a signal threshold, mimicking whether the treatment is outperforming a control. The function writes a CSV string to an in-memory buffer, then appends a summary dictionary with counts of active and inactive days. This approach is useful for testing downstream analysis code without needing real experiment data.

Common mistakes

  • Confusing `day < horizon` with `day <= horizon` — the horizon day itself is treated as post-warmup.
  • Forgetting that rows before horizon are labeled 'warmup' and not counted in active/inactive totals.
  • Using mutable default arguments when holding state — here we use a local StringIO, avoiding that issue.
  • Forgetting to reset the StringIO pointer before reading if more complex output handling is needed.

Variations

  1. Use `csv.DictWriter` with fieldnames to write dictionaries per row.
  2. Return a pandas DataFrame instead of a CSV string for easier downstream analysis.

Real-world use cases

  • Generating synthetic data to unit test an experiment analysis pipeline before real data arrives.
  • Simulating the expected labels and summary metrics for a fixed-horizon A/B test in a staging environment.
  • Creating illustrative examples for documentation or training materials on experiment evaluation.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from A/B testing & experimentation

Keep learning

Related tutorials and quizzes for this topic.

Tutorials

Quizzes