How to implement exponential backoff for LLM API calls in Python

A decorator that retries flaky LLM API calls with exponential delay, using a mock client to demonstrate the pattern.

Medium Python 3.9+ Aug 9, 2026 AI & LLM integration patterns 15 views 0 copies

exponential-backoff retries llm decorator resilience

Python code

36 lines
Python 3.9+
Run it 
import time
import random

class MockLLM:
    def call(self, prompt):
        if random.random() < 0.7:  # 70% chance of transient failure
            raise ConnectionError("API unavailable")
        return f"LLM response for: {prompt}"

def with_exponential_backoff(max_retries=5, base_delay=0.1):
    def decorator(func):
        def wrapper(*args, **kwargs):
            delay = base_delay
            for attempt in range(max_retries + 1):
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    if attempt == max_retries:
                        raise e
                    print(f"Attempt {attempt + 1} failed: {e}")
                    time.sleep(delay)
                    delay *= 2
            return None
        return wrapper
    return decorator

if __name__ == "__main__":
    random.seed(42)
    api = MockLLM()

    @with_exponential_backoff(max_retries=3, base_delay=0.05)
    def call_llm(prompt):
        return api.call(prompt)

    result = call_llm("Translate to French")
    print(f"Final result: {result}")

Output

stdout 
Attempt 1 failed: API unavailable
Attempt 2 failed: API unavailable
Final result: LLM response for: Translate to French

How it works

The decorator wraps any function and retries it when it raises an exception. Each retry waits delay seconds, then doubles it for the next attempt (delay *= 2). The max_retries parameter bounds the total attempts; the final failure re-raises the original exception so the caller can handle it. Using a MocksLLM class with random failures lets you test the retry logic without hitting a real API. The random.seed(42) ensures deterministic output for reproducible tests.

Common mistakes

  • Raising the exception inside the loop before the final attempt, which skips retries
  • Forgetting to multiply the delay, so subsequent retries don't actually back off
  • Not catching all exception types, missing transient network errors
  • Sleeping too long with a large base delay, blocking the caller thread

Variations

  1. Use `functools.wraps(func)` to preserve the original function's metadata.
  2. Add jitter: `delay = random.uniform(0, delay)` to avoid thundering herd.

Real-world use cases

  • Retrying OpenAI or Anthropic API calls that fail with 429 or 503 status codes.
  • Wrapping batch inference jobs that hit rate limits in a shared production pipeline.
  • Recovering from transient network failures when a background worker calls a hosted LLM service.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from AI & LLM integration patterns

Keep learning

Related tutorials and quizzes for this topic.

Tutorials

Quizzes