How to Simulate an Outbox Pattern with Reliable Retry in Python

This code implements a mock outbox pattern with records, delivery attempts, and retries to simulate reliable message publishing.

Medium Python 3.9+ Aug 9, 2026 Reliability & rate limiting 14 views 0 copies

Python code

50 lines
Python 3.9+
import time
import itertools

class Outbox:
    def __init__(self):
        self._records = []
        self._seq = itertools.count(1)

    def publish(self, topic, payload):
        record = {
            "id": next(self._seq),
            "topic": topic,
            "payload": payload,
            "status": "pending",
            "attempts": 0,
        }
        self._records.append(record)
        return record

    def deliver(self, mock_delivery=lambda rec: True):
        delivered = []
        for record in self._records:
            if record["status"] != "pending":
                continue
            if mock_delivery(record):
                record["status"] = "delivered"
                record["delivered_at"] = time.time()
                delivered.append(record["id"])
            else:
                record["attempts"] += 1
        return delivered

    def pending_records(self):
        return [r for r in self._records if r["status"] == "pending"]


if __name__ == "__main__":
    outbox = Outbox()
    outbox.publish("order.created", {"order_id": 1})
    outbox.publish("payment.success", {"order_id": 1})

    def flaky_delivery(record):
        return record["id"] != 2  # Simulate a temporary failure for id=2

    outbox.deliver(mock_delivery=flaky_delivery)
    retry = outbox.deliver(mock_delivery=lambda rec: True)  # Retry pending

    print("Delivered first pass:", [1] if outbox.pending_records() else [])
    print("Retry delivered:", retry)
    print("Pending after retry:", outbox.pending_records())

Output

stdout
Delivered first pass: []
Retry delivered: [2]
Pending after retry: []

How it works

The Outbox class stores publish requests as pending records with a unique ID and attempt counter. The deliver method processes all pending records, marking them as delivered on success or incrementing attempts on failure. This retry loop ensures messages are eventually delivered, emulating a reliable outbox pattern. The mock delivery function lets you simulate transient failures without external dependencies.

Common mistakes

  • Not resetting the sequence counter when reusing the outbox in tests
  • Forgetting to filter only pending records before delivery, causing duplicates
  • Assuming delivery succeeds without tracking attempts for dead-letter handling

Variations

  1. Use a database table instead of an in-memory list for durable persistence
  2. Add a max_attempts parameter to move failed records to a dead-letter queue

Real-world use cases

  • Guaranteeing message delivery before confirming a database transaction in distributed systems.
  • Persisting publish intents in a transactional outbox to ensure atomicity with business events.
  • Testing retry logic in CI pipelines by mocking delivery failures without external brokers.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Reliability & rate limiting

Related tutorials and quizzes for this topic.