Auto Rollback on Error Rate Exceeded in Python
Simulate a service that monitors a rolling window of request errors and automatically rolls back when the error rate exceeds a threshold.
Python code
42 linesimport random
import time
def simulate_requests(total_requests=1000, rollback_threshold=0.2):
"""
Simulate a service that automatically rolls back when the error rate
exceeds a threshold within a rolling window.
"""
window_size = 100
errors_seen = []
rolled_back = False
for req_num in range(1, total_requests + 1):
# Simulate error with ~10% baseline, but introduce a spike mid-way
error_rate = 0.10 if req_num < 400 else (0.30 if req_num < 700 else 0.05)
is_error = random.random() < error_rate
errors_seen.append(is_error)
# Keep only the last `window_size` requests
if len(errors_seen) > window_size:
errors_seen.pop(0)
# Check error rate in the current window
if len(errors_seen) == window_size:
current_error_rate = sum(errors_seen) / window_size
if current_error_rate > rollback_threshold and not rolled_back:
rolled_back = True
print(f"ROLLBACK triggered at request #{req_num} "
f"(window error rate: {current_error_rate:.2%})")
return # Stop processing after rollback
# Simulate processing time
time.sleep(0)
print(f"Completed {total_requests} requests without rollback "
f"(final error rate: {sum(errors_seen) / len(errors_seen):.2%})")
if __name__ == "__main__":
random.seed(123)
simulate_requests()
Output
ROLLBACK triggered at request #434 (window error rate: 22.00%)
How it works
The simulation tracks the last window_size requests and calculates the rolling error rate. If the error rate exceeds the threshold within the window, it triggers a rollback and stops processing. The error rate is simulated with a baseline of 10%, spiking to 30% mid-run to demonstrate the rollback. Using random.seed ensures reproducible output. The rolling window approach provides a smooth threshold check without resetting on each request.
Common mistakes
- Using the cumulative error rate instead of a rolling window, which masks recent spikes.
- Forgetting to seed randomness, making tests non-reproducible.
- Not clearing the window when a rollback happens, leading to repeated triggers.
Variations
- Use `collections.deque(maxlen=window_size)` for efficient rolling storage instead of a list with `pop(0)`.
- Real deployments would integrate with a circuit breaker library or service mesh.
Real-world use cases
- Automatically reverting a new microservice deployment when the 5xx rate spikes beyond SLO in a rolling window.
- Triggering a feature flag rollback in a canary release when the error ratio on a sample of traffic exceeds a threshold.
- Stopping a batch data pipeline job if the processing failure rate climbs during a window, preventing further data corruption.
Sponsored
More from Production deployment patterns
- Automate Semantic Versioning with Conventional Commits in Python medium
- Design a Data Helper for Beginners in Python easy
- Docker healthcheck CMD mock in Python easy
- Generate a Mock Artifact Version Tag in Python easy
- Generate a docker-compose.yml with mock services in Python easy
- How to Attach an SBOM to a Release in Python (Mock) easy
Keep learning
Related tutorials and quizzes for this topic.