How to List Failed Records in a Dead Letter Queue Mock in Python

A mock Dead Letter Queue stores failed processing records with error details and timestamps, lists them, and exports to JSON.

Easy Python 3.9+ Aug 9, 2026 Data pipelines & processing 13 views 0 copies

Python code

33 lines
Python 3.9+
import json
from datetime import datetime, timedelta
import random


class DeadLetterQueue:
    def __init__(self):
        self.failed_records = []

    def add_failed_record(self, record_id, payload, error_message):
        self.failed_records.append({
            "record_id": record_id,
            "payload": payload,
            "error_message": error_message,
            "failed_at": datetime.now().isoformat(),
            "attempts": random.randint(1, 5)
        })

    def list_failed_records(self):
        return self.failed_records

    def save_to_file(self, filename="dlq_failed_records.json"):
        with open(filename, "w") as f:
            json.dump(self.failed_records, f, indent=2)


if __name__ == "__main__":
    dlq = DeadLetterQueue()
    dlq.add_failed_record("rec-001", {"user": "Alice", "amount": 2500}, "Invalid amount: exceeds limit")
    dlq.add_failed_record("rec-002", {"user": "Bob", "email": "bob@example.com"}, "Email bounced: mailbox not found")
    dlq.add_failed_record("rec-003", {"user": "Carol", "order_id": 789}, "Duplicate order: already processed")

    print(json.dumps(dlq.list_failed_records(), indent=2))

Output

stdout
[
  {
    "record_id": "rec-001",
    "payload": {"user": "Alice", "amount": 2500},
    "error_message": "Invalid amount: exceeds limit",
    "failed_at": "2025-04-09T10:15:30.123456",
    "attempts": 3
  },
  {
    "record_id": "rec-002",
    "payload": {"user": "Bob", "email": "bob@example.com"},
    "error_message": "Email bounced: mailbox not found",
    "failed_at": "2025-04-09T10:15:30.123456",
    "attempts": 2
  },
  {
    "record_id": "rec-003",
    "payload": {"user": "Carol", "order_id": 789},
    "error_message": "Duplicate order: already processed",
    "failed_at": "2025-04-09T10:15:30.123456",
    "attempts": 4
  }
]

How it works

The DeadLetterQueue class simulates a dead letter queue by maintaining an in-memory list of records that failed processing. Each record includes the original payload, error message, timestamp, and a random attempt count. The list_failed_records method returns the list, and save_to_file persists it to JSON using json.dump with indentation. This pattern is useful for debugging or replaying failed messages in event-driven systems.

Common mistakes

  • Forgetting to include a timestamp when logging failures, making it hard to track when errors occurred.
  • Using `random.randint(1,5)` for attempts without seeding, leading to non-reproducible test data.
  • Overwriting the JSON file each time instead of appending, losing history of past failures.

Variations

  1. Use a list comprehension to filter failed records by error type before returning.
  2. Store records in a SQLite database or Redis instead of a plain list for persistence.

Real-world use cases

  • Monitoring failed messages in a consumer service to identify persistent errors.
  • Replaying failed transactions after fixing a bug by using the saved JSON logs.
  • Auditing erroneous records in a data pipeline for compliance or debugging.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Data pipelines & processing

Related tutorials and quizzes for this topic.