How to Send Messages to a Dead Letter Queue in Python
Simulates a poison message queue that retries failed messages up to a limit before moving them to a dead letter queue.
Python code
40 linesimport json
class PoisonMessageQueue:
def __init__(self, max_retries=3):
self.dlq = []
self.max_retries = max_retries
self.processed_count = 0
self.failed_count = 0
def process_message(self, message_body):
if "poison" in message_body:
self.failed_count += 1
retries = message_body.get("retries", 0)
message_body["retries"] = retries + 1
if retries >= self.max_retries - 1:
self.dlq.append(message_body)
return "sent_to_dlq"
return "will_retry"
self.processed_count += 1
return "processed"
def summary(self):
return {
"processed": self.processed_count,
"failed": self.failed_count,
"dlq_length": len(self.dlq)
}
if __name__ == "__main__":
queue_service = PoisonMessageQueue(max_retries=3)
messages = [
{"id": 1, "content": "normal message"},
{"id": 2, "content": "poison message", "retries": 0},
{"id": 3, "content": "poison message", "retries": 1},
{"id": 4, "content": "poison message", "retries": 2}
]
for msg in messages:
result = queue_service.process_message(msg)
print(f"Message {msg['id']}: {result}")
print("Summary:", json.dumps(queue_service.summary()))
Output
Message 1: processed
Message 2: will_retry
Message 3: will_retry
Message 4: sent_to_dlq
Summary: {"processed": 1, "failed": 3, "dlq_length": 1}
How it works
This mock tracks retries per message inside the message body, incrementing the 'retries' key each time a poison message is encountered. Once the retry count reaches max_retries - 1, the message is moved to a separate DLQ list, preventing infinite reprocessing. The processed_count and failed_count provide observability into the queue's health. This pattern mirrors real-world message brokers like RabbitMQ or Kafka, where poison messages are isolated for manual inspection.
Common mistakes
- Resetting the retry counter incorrectly between attempts
- Forgetting to persist retry metadata if messages are lost
- Not handling messages that lack the 'retries' key gracefully
Variations
- Use a dictionary to store retry counts externally instead of mutating the message body
- Implement exponential backoff delays between retries
Real-world use cases
- Message queue workers that need to quarantine malformed payloads instead of crashing the consumer.
- Event processing systems where dead-lettering prevents data loss for downstream analytics.
- Microservices that retry transient failures before escalating to a DLQ for operational review.
Sponsored
More from Reliability & rate limiting
- At Least Once with Idempotent Consumer in Python medium
- Build a Rate Limiter Decorator in Python easy
- Build a queue-based admission control system in Python easy
- Chaos Inject Random Failures in Python easy
- Circuit breaker failure threshold count in Python medium
- Exactly Once Processing Dedupe Mock in Python easy
Keep learning
Related tutorials and quizzes for this topic.