How to Implement a Dead Letter Queue Replay in Python
A mock Dead Letter Queue that stores failed messages with retry attempts and replays them with a simple retry counter.
Python code
39 linesimport json
from collections import deque
class DeadLetterQueue:
def __init__(self):
self.messages = deque()
def add_message(self, message_id, payload, attempts=3):
"""Add a message to the DLQ with retry metadata."""
self.messages.append({
"id": message_id,
"payload": payload,
"attempts_left": attempts
})
def replay(self, message_id):
"""Replay a specific message, decrementing its attempts."""
for msg in self.messages:
if msg["id"] == message_id:
if msg["attempts_left"] <= 0:
print(f"Message {message_id} exhausted (0 retries left)")
return None
msg["attempts_left"] -= 1
payload = msg["payload"]
print(f"Replayed {message_id}: {json.dumps(payload)}")
return payload
print(f"Message {message_id} not found")
return None
if __name__ == "__main__":
dlq = DeadLetterQueue()
dlq.add_message(1, {"event": "payment_failed", "amount": 99.50})
dlq.add_message(2, {"event": "email_bounced", "address": "user@example.com"})
dlq.replay(1)
dlq.replay(2)
dlq.replay(1)
dlq.replay(1)
dlq.replay(99)
Output
Replayed 1: {"event": "payment_failed", "amount": 99.5}
Replayed 2: {"event": "email_bounced", "address": "user@example.com"}
Replayed 1: {"event": "payment_failed", "amount": 99.5}
Message 1 exhausted (0 retries left)
Message 99 not found
How it works
This mock DLQ uses a deque to store messages in insertion order, allowing efficient appends and iteration. Each message carries an attempts_left field that is decremented on every replay call, simulating a retry budget. When the budget reaches zero, the message is considered exhausted and cannot be replayed further. The replay method simulates the action of sending the message back to a consumer and returns the payload for further processing. This pattern is useful for modeling retry policies without external dependencies.
Common mistakes
- Not decrementing attempts before checking if it can be replayed, causing off-by-one errors
- Forgetting to mark exhausted messages as permanently failed or removing them
- Assuming message order is guaranteed when using plain lists instead of deque
- Not using `json.dumps` to format payloads for readable logs
Variations
- Use a dictionary keyed by message ID for O(1) lookups instead of scanning all messages
- Add a `pending` flag to track whether a message is available for replay
Real-world use cases
- Replaying failed Kafka events after a downstream service recovers from an outage.
- Retrying webhook deliveries that temporarily failed due to rate limits or timeouts.
- Requeueing failed API requests in a job queue with a configurable max retry count.
Sponsored
More from Reliability & rate limiting
- At Least Once with Idempotent Consumer in Python medium
- Build a Rate Limiter Decorator in Python easy
- Build a queue-based admission control system in Python easy
- Chaos Inject Random Failures in Python easy
- Circuit breaker failure threshold count in Python medium
- Exactly Once Processing Dedupe Mock in Python easy
Keep learning
Related tutorials and quizzes for this topic.