How to retry idempotent operations with a mock in Python

Wrap a flaky idempotent operation in a retry loop with exponential backoff, and use unittest.mock to deterministically test the str's behavior.

Medium Python 3.9+ Aug 9, 2026 Reliability & rate limiting 14 views 0 copies

Python code

45 lines
Python 3.9+
import random
import time
from unittest.mock import Mock


def idempotent_operation(value):
    """Simulate an idempotent operation that sometimes fails."""
    if random.random() < 0.6:  # 60% failure rate
        raise ConnectionError("Temporary failure")
    return value * 2


def retry_with_backoff(operation, max_attempts=3, base_delay=0.1):
    """Retry an idempotent operation with exponential backoff."""
    for attempt in range(max_attempts):
        try:
            result = operation()
            print(f"Attempt {attempt + 1}: Success -> {result}")
            return result
        except ConnectionError as e:
            print(f"Attempt {attempt + 1}: Failed ({e})")
            if attempt < max_attempts - 1:
                time.sleep(base_delay * (2 ** attempt))
    print("All attempts failed")
    return None


if __name__ == "__main__":
    # Mock the operation to control failure/success deterministically
    operation_mock = Mock(side_effect=[ConnectionError("Down"), ConnectionError("Down"), 42])

    # Patch random to make the real function predictable for comparison
    original_random = random.random
    random.random = lambda: 1.0  # Force failure in real function

    print("--- Using mocked operation (deterministic) ---")
    result = retry_with_backoff(operation_mock)
    print(f"Final result: {result}")

    print("\n--- Using real operation patched to always fail ---")
    result2 = retry_with_backoff(lambda: idempotent_operation(21))
    print(f"Final result: {result2}")

    # Restore original random
    random.random = original_random

Output

stdout
--- Using mocked operation (deterministic) ---
Attempt 1: Failed (Down)
Attempt 2: Failed (Down)
Attempt 3: Success -> 42
Final result: 42

--- Using real operation patched to always fail ---
Attempt 1: Failed (Temporary failure)
Attempt 2: Failed (Temporary failure)
Attempt 3: Failed (Temporary failure)
All attempts failed
Final result: None

How it works

The retry_with_backoff function loops up to max_attempts, catching ConnectionError and sleeping with exponential delay (base_delay * 2**attempt) between tries. Using unittest.mock.Mock(side_effect=[...]) lets you provide a fixed sequence of outcomes, making the retry logic fully deterministic — no flaky tests. Patching random.random forces the real operation to always fail so you can verify the exhausted-retry path. The function returns the first successful result or None after all attempts, and every attempt prints its status for clear tracing.

Common mistakes

  • Retrying operations that are not idempotent, causing duplicate side effects.
  • Forgetting to restore mocked globals like `random.random` after the test — use `patch` or try/finally.
  • Using a fixed delay instead of exponential backoff, which can hammer the service under load.
  • Swallowing all exceptions instead of catching only transient ones like `ConnectionError`.

Variations

  1. Use `functools.partial` to pass arguments to the operation while keeping the retry signature simple.
  2. Add jitter (random sleep offset) to backoff to avoid thundering-herd retries in distributed systems.

Real-world use cases

  • Retrying idempotent API calls (e.g., POST with an idempotency key) when the network briefly drops.
  • Reconnecting to a database or queue after a temporary connection loss during a batch job.
  • Testing retry logic in CI without relying on a flaky external service by mocking failure sequences.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Reliability & rate limiting

Related tutorials and quizzes for this topic.