Implement the Transactional Outbox Pattern with SQLite in Python

A Python implementation of the transactional outbox pattern using SQLite, ensuring atomic writes of order data and outbox events in a single transaction while supporting reliable message publishing and consumption.

Medium Python 3.9+ Aug 9, 2026 Streaming & messaging 17 views 0 copies

Python code

75 lines
Python 3.9+
import sqlite3
from dataclasses import dataclass
from datetime import datetime, timezone
import json

@dataclass
class Order:
    order_id: str
    amount: float
    status: str

class TransactionalOutbox:
    def __init__(self, db_path=":memory:"):
        self.conn = sqlite3.connect(db_path)
        self._create_tables()

    def _create_tables(self):
        with self.conn:
            self.conn.execute("""
                CREATE TABLE IF NOT EXISTS orders (
                    id TEXT PRIMARY KEY,
                    amount REAL,
                    status TEXT
                )
            """)
            self.conn.execute("""
                CREATE TABLE IF NOT EXISTS outbox (
                    id INTEGER PRIMARY KEY AUTOINCREMENT,
                    payload TEXT,
                    created_at TEXT,
                    processed INTEGER DEFAULT 0
                )
            """)

    def create_order(self, order: Order):
        with self.conn:
            # Insert order in the same transaction as outbox event
            self.conn.execute(
                "INSERT INTO orders VALUES (?, ?, ?)",
                (order.order_id, order.amount, order.status)
            )
            event = {
                "type": "order_created",
                "order_id": order.order_id,
                "amount": order.amount
            }
            self.conn.execute(
                "INSERT INTO outbox (payload, created_at) VALUES (?, ?)",
                (json.dumps(event), datetime.now(timezone.utc).isoformat())
            )

    def poll_outbox(self):
        with self.conn:
            events = self.conn.execute(
                "SELECT id, payload FROM outbox WHERE processed = 0 LIMIT 10"
            ).fetchall()
            for event_id, payload in events:
                print(f"Publishing: {payload}")
                self.conn.execute(
                    "UPDATE outbox SET processed = 1 WHERE id = ?",
                    (event_id,)
                )

if __name__ == "__main__":
    outbox = TransactionalOutbox()
    outbox.create_order(Order("ORD-1", 99.50, "pending"))
    outbox.create_order(Order("ORD-2", 250.00, "pending"))
    
    print("Outbox contents before processing:")
    events = outbox.conn.execute("SELECT payload FROM outbox").fetchall()
    for row in events:
        print(row[0])
    
    print("\nPolling outbox:")
    outbox.poll_outbox()

Output

stdout
Outbox contents before processing:
{"type": "order_created", "order_id": "ORD-1", "amount": 99.5}
{"type": "order_created", "order_id": "ORD-2", "amount": 250.0}

Polling outbox:
Publishing: {"type": "order_created", "order_id": "ORD-1", "amount": 99.5}
Publishing: {"type": "order_created", "order_id": "ORD-2", "amount": 250.0}

How it works

The create_order method wraps the order insert and outbox event insert in a single SQLite transaction using the with self.conn context manager, guaranteeing that either both succeed or neither is persisted. The outbox table uses an AUTOINCREMENT primary key to maintain event ordering and a processed flag to track which events have been published. The poll_outbox method retrieves unprocessed events in batches and flips their processed flag to prevent duplicate delivery. This pattern decouples the database write from the external message publishing, ensuring no events are lost if the broker fails.

Common mistakes

  • Calling commit() manually when using the 'with self.conn' context manager, which already auto-commits on success.
  • Forgetting to mark events as processed after publishing, causing duplicate messages on the next poll.
  • Publishing the message before the database transaction is committed, breaking atomicity guarantees.
  • Using local timestamps instead of timezone-aware UTC datetimes for event consistency across distributed systems.

Variations

  1. Use a dedicated outbox table with a JSONB column in PostgreSQL instead of SQLite for production environments.
  2. Replace the sequential poll loop with a background worker using asyncio or cron jobs for continuous event publishing.

Real-world use cases

  • Ensuring order events are never lost when publishing to Kafka or RabbitMQ in an e-commerce checkout flow.
  • Syncing user profile changes to a search index or analytics platform without duplicating updates.
  • Reliably emitting domain events from a monolith to trigger downstream services like email notifications.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Streaming & messaging

Related tutorials and quizzes for this topic.