Auto Rollback on Error Rate Exceeded in Python

Simulate a service that monitors a rolling window of request errors and automatically rolls back when the error rate exceeds a threshold.

Medium Python 3.9+ Aug 9, 2026 Production deployment patterns 15 views 0 copies

Python code

42 lines
Python 3.9+
import random
import time


def simulate_requests(total_requests=1000, rollback_threshold=0.2):
    """
    Simulate a service that automatically rolls back when the error rate
    exceeds a threshold within a rolling window.
    """
    window_size = 100
    errors_seen = []
    rolled_back = False

    for req_num in range(1, total_requests + 1):
        # Simulate error with ~10% baseline, but introduce a spike mid-way
        error_rate = 0.10 if req_num < 400 else (0.30 if req_num < 700 else 0.05)
        is_error = random.random() < error_rate
        errors_seen.append(is_error)

        # Keep only the last `window_size` requests
        if len(errors_seen) > window_size:
            errors_seen.pop(0)

        # Check error rate in the current window
        if len(errors_seen) == window_size:
            current_error_rate = sum(errors_seen) / window_size
            if current_error_rate > rollback_threshold and not rolled_back:
                rolled_back = True
                print(f"ROLLBACK triggered at request #{req_num} "
                      f"(window error rate: {current_error_rate:.2%})")
                return  # Stop processing after rollback

        # Simulate processing time
        time.sleep(0)

    print(f"Completed {total_requests} requests without rollback "
          f"(final error rate: {sum(errors_seen) / len(errors_seen):.2%})")


if __name__ == "__main__":
    random.seed(123)
    simulate_requests()

Output

stdout
ROLLBACK triggered at request #434 (window error rate: 22.00%)

How it works

The simulation tracks the last window_size requests and calculates the rolling error rate. If the error rate exceeds the threshold within the window, it triggers a rollback and stops processing. The error rate is simulated with a baseline of 10%, spiking to 30% mid-run to demonstrate the rollback. Using random.seed ensures reproducible output. The rolling window approach provides a smooth threshold check without resetting on each request.

Common mistakes

  • Using the cumulative error rate instead of a rolling window, which masks recent spikes.
  • Forgetting to seed randomness, making tests non-reproducible.
  • Not clearing the window when a rollback happens, leading to repeated triggers.

Variations

  1. Use `collections.deque(maxlen=window_size)` for efficient rolling storage instead of a list with `pop(0)`.
  2. Real deployments would integrate with a circuit breaker library or service mesh.

Real-world use cases

  • Automatically reverting a new microservice deployment when the 5xx rate spikes beyond SLO in a rolling window.
  • Triggering a feature flag rollback in a canary release when the error ratio on a sample of traffic exceeds a threshold.
  • Stopping a batch data pipeline job if the processing failure rate climbs during a window, preventing further data corruption.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Production deployment patterns

Related tutorials and quizzes for this topic.