How to List Failed Records in a Dead Letter Queue Mock in Python
A mock Dead Letter Queue stores failed processing records with error details and timestamps, lists them, and exports to JSON.
Python code
33 linesimport json
from datetime import datetime, timedelta
import random
class DeadLetterQueue:
def __init__(self):
self.failed_records = []
def add_failed_record(self, record_id, payload, error_message):
self.failed_records.append({
"record_id": record_id,
"payload": payload,
"error_message": error_message,
"failed_at": datetime.now().isoformat(),
"attempts": random.randint(1, 5)
})
def list_failed_records(self):
return self.failed_records
def save_to_file(self, filename="dlq_failed_records.json"):
with open(filename, "w") as f:
json.dump(self.failed_records, f, indent=2)
if __name__ == "__main__":
dlq = DeadLetterQueue()
dlq.add_failed_record("rec-001", {"user": "Alice", "amount": 2500}, "Invalid amount: exceeds limit")
dlq.add_failed_record("rec-002", {"user": "Bob", "email": "bob@example.com"}, "Email bounced: mailbox not found")
dlq.add_failed_record("rec-003", {"user": "Carol", "order_id": 789}, "Duplicate order: already processed")
print(json.dumps(dlq.list_failed_records(), indent=2))
Output
[
{
"record_id": "rec-001",
"payload": {"user": "Alice", "amount": 2500},
"error_message": "Invalid amount: exceeds limit",
"failed_at": "2025-04-09T10:15:30.123456",
"attempts": 3
},
{
"record_id": "rec-002",
"payload": {"user": "Bob", "email": "bob@example.com"},
"error_message": "Email bounced: mailbox not found",
"failed_at": "2025-04-09T10:15:30.123456",
"attempts": 2
},
{
"record_id": "rec-003",
"payload": {"user": "Carol", "order_id": 789},
"error_message": "Duplicate order: already processed",
"failed_at": "2025-04-09T10:15:30.123456",
"attempts": 4
}
]
How it works
The DeadLetterQueue class simulates a dead letter queue by maintaining an in-memory list of records that failed processing. Each record includes the original payload, error message, timestamp, and a random attempt count. The list_failed_records method returns the list, and save_to_file persists it to JSON using json.dump with indentation. This pattern is useful for debugging or replaying failed messages in event-driven systems.
Common mistakes
- Forgetting to include a timestamp when logging failures, making it hard to track when errors occurred.
- Using `random.randint(1,5)` for attempts without seeding, leading to non-reproducible test data.
- Overwriting the JSON file each time instead of appending, losing history of past failures.
Variations
- Use a list comprehension to filter failed records by error type before returning.
- Store records in a SQLite database or Redis instead of a plain list for persistence.
Real-world use cases
- Monitoring failed messages in a consumer service to identify persistent errors.
- Replaying failed transactions after fixing a bug by using the saved JSON logs.
- Auditing erroneous records in a data pipeline for compliance or debugging.
Sponsored
More from Data pipelines & processing
- Add a UUID Surrogate Key to Each Row in a CSV with Python easy
- Attach Source File Metadata to Records in Python easy
- Build a Python Utility That Detects Duplicate Records Across Multiple Excel Sheets medium
- Check Null Rate Threshold in PySpark DataFrame medium
- Count Records Processed per Category in Python easy
- Create Data Helper Functions in Python for Beginners easy
Keep learning
Related tutorials and quizzes for this topic.