Outbox pattern reliable publish in Python with SQLite
Implements a transactional outbox with SQLite, ensuring reliable message publishing by storing events in the same DB transaction as business changes.
Python code
55 linesimport sqlite3
from contextlib import contextmanager
from datetime import datetime, timezone
class Outbox:
def __init__(self, db_path=":memory:"):
self.conn = sqlite3.connect(db_path)
self.conn.execute("""
CREATE TABLE IF NOT EXISTS outbox (
id INTEGER PRIMARY KEY AUTOINCREMENT,
payload TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'pending',
created_at TEXT NOT NULL
)
""")
self.conn.commit()
@contextmanager
def transaction(self):
try:
yield
self.conn.commit()
except Exception:
self.conn.rollback()
raise
def enqueue(self, payload: str) -> int:
with self.transaction():
cur = self.conn.execute(
"INSERT INTO outbox (payload, created_at) VALUES (?, ?)",
(payload, datetime.now(timezone.utc).isoformat()),
)
return cur.lastrowid
def publish_pending(self) -> list[tuple[int, str]]:
with self.transaction():
rows = self.conn.execute(
"SELECT id, payload FROM outbox WHERE status = 'pending'"
).fetchall()
self.conn.execute(
"UPDATE outbox SET status = 'published' WHERE status = 'pending'"
)
return [(row_id, payload) for row_id, payload in rows]
if __name__ == "__main__":
outbox = Outbox()
outbox.enqueue('{"type": "order.created", "id": 1}')
outbox.enqueue('{"type": "order.created", "id": 2}')
published = outbox.publish_pending()
print("Published messages:", published)
print("Remaining pending:", outbox.conn.execute(
"SELECT COUNT(*) FROM outbox WHERE status = 'pending'"
).fetchone()[0])
Output
Published messages: [(1, '{"type": "order.created", "id": 1}'), (2, '{"type": "order.created", "id": 2}')]
Remaining pending: 0
How it works
The outbox pattern guarantees that every business change and its corresponding message are committed atomically. By inserting the event into the same SQLite transaction as the business operation, you avoid the dual-write problem. The transaction context manager commits on success and rolls back on exception, keeping the table consistent. publish_pending reads pending rows and marks them published in one transaction, so a crash mid-publish doesn't lose messages. This simulates the reliable core of an outbox before adding a message broker.
Common mistakes
- Committing the business change and outbox insert separately, losing atomicity
- Forgetting to mark messages as published after sending, causing duplicates
- Not using a transaction for the publish-read-update cycle, risking partial processing
Variations
- Use PostgreSQL with a trigger or procedure to automate the outbox insert
- Add a retry worker that picks up stuck pending messages after a timeout
Real-world use cases
- E-commerce order creation: persist order and 'order.created' event atomically to avoid missing notifications.
- Payment processing: record payment and 'payment.succeeded' event in one transaction for audit and downstream sync.
- User registration: store user and 'user.signed_up' event together so a welcome email always fires exactly once.
Sponsored
More from System design patterns
- Build a BFF (Backend for Frontend) Mock Aggregator in Python medium
- Builder pattern for mocking complex objects in Python easy
- Circuit Breaker Pattern in Python: Closed, Open, and Half-Open States medium
- Create a Data Helper Class in Python easy
- Domain Driven Design Aggregate Root Example in Python medium
- Facade Pattern in Python with Mock Simplification medium
Keep learning
Related tutorials and quizzes for this topic.