How to retry idempotent operations with a mock in Python
Wrap a flaky idempotent operation in a retry loop with exponential backoff, and use unittest.mock to deterministically test the str's behavior.
Python code
45 linesimport random
import time
from unittest.mock import Mock
def idempotent_operation(value):
"""Simulate an idempotent operation that sometimes fails."""
if random.random() < 0.6: # 60% failure rate
raise ConnectionError("Temporary failure")
return value * 2
def retry_with_backoff(operation, max_attempts=3, base_delay=0.1):
"""Retry an idempotent operation with exponential backoff."""
for attempt in range(max_attempts):
try:
result = operation()
print(f"Attempt {attempt + 1}: Success -> {result}")
return result
except ConnectionError as e:
print(f"Attempt {attempt + 1}: Failed ({e})")
if attempt < max_attempts - 1:
time.sleep(base_delay * (2 ** attempt))
print("All attempts failed")
return None
if __name__ == "__main__":
# Mock the operation to control failure/success deterministically
operation_mock = Mock(side_effect=[ConnectionError("Down"), ConnectionError("Down"), 42])
# Patch random to make the real function predictable for comparison
original_random = random.random
random.random = lambda: 1.0 # Force failure in real function
print("--- Using mocked operation (deterministic) ---")
result = retry_with_backoff(operation_mock)
print(f"Final result: {result}")
print("\n--- Using real operation patched to always fail ---")
result2 = retry_with_backoff(lambda: idempotent_operation(21))
print(f"Final result: {result2}")
# Restore original random
random.random = original_random
Output
--- Using mocked operation (deterministic) ---
Attempt 1: Failed (Down)
Attempt 2: Failed (Down)
Attempt 3: Success -> 42
Final result: 42
--- Using real operation patched to always fail ---
Attempt 1: Failed (Temporary failure)
Attempt 2: Failed (Temporary failure)
Attempt 3: Failed (Temporary failure)
All attempts failed
Final result: None
How it works
The retry_with_backoff function loops up to max_attempts, catching ConnectionError and sleeping with exponential delay (base_delay * 2**attempt) between tries. Using unittest.mock.Mock(side_effect=[...]) lets you provide a fixed sequence of outcomes, making the retry logic fully deterministic — no flaky tests. Patching random.random forces the real operation to always fail so you can verify the exhausted-retry path. The function returns the first successful result or None after all attempts, and every attempt prints its status for clear tracing.
Common mistakes
- Retrying operations that are not idempotent, causing duplicate side effects.
- Forgetting to restore mocked globals like `random.random` after the test — use `patch` or try/finally.
- Using a fixed delay instead of exponential backoff, which can hammer the service under load.
- Swallowing all exceptions instead of catching only transient ones like `ConnectionError`.
Variations
- Use `functools.partial` to pass arguments to the operation while keeping the retry signature simple.
- Add jitter (random sleep offset) to backoff to avoid thundering-herd retries in distributed systems.
Real-world use cases
- Retrying idempotent API calls (e.g., POST with an idempotency key) when the network briefly drops.
- Reconnecting to a database or queue after a temporary connection loss during a batch job.
- Testing retry logic in CI without relying on a flaky external service by mocking failure sequences.
Sponsored
More from Reliability & rate limiting
- At Least Once with Idempotent Consumer in Python medium
- Build a Rate Limiter Decorator in Python easy
- Build a queue-based admission control system in Python easy
- Chaos Inject Random Failures in Python easy
- Circuit breaker failure threshold count in Python medium
- Exactly Once Processing Dedupe Mock in Python easy
Keep learning
Related tutorials and quizzes for this topic.