Outbox pattern reliable publish in Python with SQLite

Implements a transactional outbox with SQLite, ensuring reliable message publishing by storing events in the same DB transaction as business changes.

Medium Python 3.9+ Aug 9, 2026 System design patterns 11 views 0 copies

Python code

55 lines
Python 3.9+
import sqlite3
from contextlib import contextmanager
from datetime import datetime, timezone


class Outbox:
    def __init__(self, db_path=":memory:"):
        self.conn = sqlite3.connect(db_path)
        self.conn.execute("""
            CREATE TABLE IF NOT EXISTS outbox (
                id INTEGER PRIMARY KEY AUTOINCREMENT,
                payload TEXT NOT NULL,
                status TEXT NOT NULL DEFAULT 'pending',
                created_at TEXT NOT NULL
            )
        """)
        self.conn.commit()

    @contextmanager
    def transaction(self):
        try:
            yield
            self.conn.commit()
        except Exception:
            self.conn.rollback()
            raise

    def enqueue(self, payload: str) -> int:
        with self.transaction():
            cur = self.conn.execute(
                "INSERT INTO outbox (payload, created_at) VALUES (?, ?)",
                (payload, datetime.now(timezone.utc).isoformat()),
            )
            return cur.lastrowid

    def publish_pending(self) -> list[tuple[int, str]]:
        with self.transaction():
            rows = self.conn.execute(
                "SELECT id, payload FROM outbox WHERE status = 'pending'"
            ).fetchall()
            self.conn.execute(
                "UPDATE outbox SET status = 'published' WHERE status = 'pending'"
            )
            return [(row_id, payload) for row_id, payload in rows]


if __name__ == "__main__":
    outbox = Outbox()
    outbox.enqueue('{"type": "order.created", "id": 1}')
    outbox.enqueue('{"type": "order.created", "id": 2}')
    published = outbox.publish_pending()
    print("Published messages:", published)
    print("Remaining pending:", outbox.conn.execute(
        "SELECT COUNT(*) FROM outbox WHERE status = 'pending'"
    ).fetchone()[0])

Output

stdout
Published messages: [(1, '{"type": "order.created", "id": 1}'), (2, '{"type": "order.created", "id": 2}')]
Remaining pending: 0

How it works

The outbox pattern guarantees that every business change and its corresponding message are committed atomically. By inserting the event into the same SQLite transaction as the business operation, you avoid the dual-write problem. The transaction context manager commits on success and rolls back on exception, keeping the table consistent. publish_pending reads pending rows and marks them published in one transaction, so a crash mid-publish doesn't lose messages. This simulates the reliable core of an outbox before adding a message broker.

Common mistakes

  • Committing the business change and outbox insert separately, losing atomicity
  • Forgetting to mark messages as published after sending, causing duplicates
  • Not using a transaction for the publish-read-update cycle, risking partial processing

Variations

  1. Use PostgreSQL with a trigger or procedure to automate the outbox insert
  2. Add a retry worker that picks up stuck pending messages after a timeout

Real-world use cases

  • E-commerce order creation: persist order and 'order.created' event atomically to avoid missing notifications.
  • Payment processing: record payment and 'payment.succeeded' event in one transaction for audit and downstream sync.
  • User registration: store user and 'user.signed_up' event together so a welcome email always fires exactly once.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from System design patterns

Related tutorials and quizzes for this topic.