Use durable queues for crash safety

Imagine a customer submits a critical order at 11:59 PM, your order-processing service crashes seconds later, and that order vanishes forever. In-memory queues make this scenario all too real: messages live only as long as the process does. Production systems need durable queues — queues that persist messages to disk or another stable store so they survive crashes, restarts, and network partitions. This lesson shows you how durable queues work, how to use them in Python, and how to choose the right approach for your system.

The problem this lesson solves

When you build a messaging pipeline, you're often tempted to use a simple in-memory queue — a Python queue.Queue, a list, or a lightweight library like asyncio.Queue. These are fast and easy, but they share a fatal flaw: all messages are lost if the process crashes or restarts. A crash can happen for many reasons: a power outage, an unhandled exception, a forced deployment, or an OOM kill by the OS. In a real system, losing messages means losing money, data, or trust. Durable queues solve this by persisting messages to a reliable storage medium before acknowledging receipt. They turn a fragile in-memory pipeline into a crash-safe backbone.

Core concept / mental model

Think of a durable queue as a safe deposit box with a log book. Each message is like a package placed in the box. The box itself is the storage — it could be a database table, a file on disk, or a specialized message broker. The log book is the metadata that tracks which messages have been delivered, which are being processed, and which have been completed. When a consumer crashes mid-processing, the log book still records that the message was taken but not completed. On restart, the queue can redeliver that message, ensuring no data is lost.

Key terms you'll see throughout this lesson:

• Durability: The queue survives process crashes and restarts.
• Durability vs. persistence: Persistence means data survives a process restart; durability is the stronger guarantee that data survives a crash — even a sudden kill — without corruption.
• Acknowledgment (ack): A consumer tells the queue it has successfully finished a message. Until ack, the message is not removed from the queue.
• At-least-once delivery: A common guarantee with durable queues — messages may be delivered more than once, but never lost.

A useful mental picture: your in-memory queue is a whiteboard. When the power goes out, everything on the whiteboard is gone. A durable queue is a paper logbook with a whiteboard on top. You write each entry in the logbook before you write it on the whiteboard. When power returns, you can recreate the whiteboard from the logbook.

How it works step by step

To make a queue durable, you need to follow a few key steps in order. Here's the standard flow:

1. Persist the message before acknowledging receipt. When a producer sends a message, the queue writes it to a durable store (e.g., disk, database) before sending an OK back. This prevents message loss if the queue process crashes right after accepting the message.
2. Track message state. The queue maintains a state for each message: pending, in-flight, or completed. This is essential for crash recovery.
3. Deliver messages to consumers with an acknowledgment contract. The consumer must explicitly tell the queue when it has finished processing a message. If it crashes before sending the ack, the queue will redeliver the message (at-least-once delivery).
4. Clear a message only after a successful ack. Once the consumer acks, the queue can safely remove the message from the durable store.
5. Recover after a crash. On startup, the queue scans its durable store for any pending or in-flight messages and restores them to the queue. This ensures no message is lost.

Pro tip: Durability is a trade-off with throughput. Writing every message to disk before acknowledging adds latency. Many brokers let you tune durability (e.g., RabbitMQ's persistent messages, Kafka's acks=all). Choose the level that matches your business requirements — not every message needs full durability.

Hands-on walkthrough

Let's build a simple durable queue using SQLite in Python. SQLite is a great choice for local prototypes or small services because it's a full database with ACID compliance and zero external dependencies. We'll create a queue class that persists messages to a SQLite table.

First, install the required packages (standard library only):

# No external packages needed — we'll use sqlite3 and threading

Here's the core queue class:

import sqlite3
import time
import threading
from contextlib import closing

class DurableQueue:
    def __init__(self, db_path):
        self.db_path = db_path
        self._init_db()
        self._lock = threading.Lock()

    def _init_db(self):
        with closing(sqlite3.connect(self.db_path)) as conn:
            conn.execute("""
                CREATE TABLE IF NOT EXISTS messages (
                    id INTEGER PRIMARY KEY AUTOINCREMENT,
                    payload TEXT NOT NULL,
                    status TEXT DEFAULT 'pending'
                )
            """)
            conn.commit()

    def enqueue(self, payload):
        """Persist the message before returning."""
        with self._lock:
            with closing(sqlite3.connect(self.db_path)) as conn:
                conn.execute("INSERT INTO messages (payload) VALUES (?)", (payload,))
                conn.commit()

    def dequeue(self):
        """Get a pending message and mark it in-flight."""
        with self._lock:
            with closing(sqlite3.connect(self.db_path)) as conn:
                # Atomically pick a pending message
                cur = conn.execute(
                    """
                    UPDATE messages SET status='in_flight'
                    WHERE id = (SELECT id FROM messages WHERE status='pending' ORDER BY id LIMIT 1)
                    RETURNING id, payload
                    """
                )
                row = cur.fetchone()
                conn.commit()
                return row

    def acknowledge(self, msg_id):
        """Mark message as completed."""
        with self._lock:
            with closing(sqlite3.connect(self.db_path)) as conn:
                conn.execute("UPDATE messages SET status='done' WHERE id=?", (msg_id,))
                conn.commit()

    def recover(self):
        """Reset in-flight messages to pending (for crash recovery)."""
        with self._lock:
            with closing(sqlite3.connect(self.db_path)) as conn:
                conn.execute("UPDATE messages SET status='pending' WHERE status='in_flight'")
                conn.commit()

    def count(self, status=None):
        with closing(sqlite3.connect(self.db_path)) as conn:
            if status:
                row = conn.execute("SELECT COUNT(*) FROM messages WHERE status=?", (status,)).fetchone()
            else:
                row = conn.execute("SELECT COUNT(*) FROM messages").fetchone()
            return row[0]

Now let's simulate a crash scenario to prove durability.

# Create a queue and enqueue messages
q = DurableQueue('queue.db')
q.enqueue('order-123')
q.enqueue('invoice-456')

# Simulate a crash: the process dies without acks
print("Messages in queue:", q.count())
print("In-flight:", q.count('in_flight'))

# Restart the queue (simulate by creating a new instance)
q2 = DurableQueue('queue.db')
q2.recover()  # reset in-flight to pending
print("After recovery, pending:", q2.count('pending'))

When you run this, you'll see:

Messages in queue: 2
In-flight: 0
After recovery, pending: 2

In a real failure, if a consumer dequeues a message and crashes before acknowledging, the message would be in_flight. On restart, calling recover() moves it back to pending, so it is processed again. That's the essence of crash safety.

Let's also test the full processing loop with an intentional crash (simulated by skipping the ack):

# Dequeue one message but never ack
msg = q2.dequeue()
print("Dequeued:", msg)

# Simulate a crash right after dequeue
import os
# In a real crash, the process would exit. Here we just don't ack
# On restart, recover in-flight messages

q3 = DurableQueue('queue.db')
q3.recover()
print("Recovered pending messages:", q3.count('pending'))

Output:

Dequeued: (1, 'order-123')
Recovered pending messages: 2

The message that was dequeued but not acked is back in the queue. This is at-least-once delivery — you get it at least once, but your consumer must be idempotent to handle duplicates.

Pro tip: Always design your consumers to be idempotent — they should handle processing the same message twice without side effects. Durable queues guarantee at-least-once delivery, not exactly-once.

Compare options / when to choose what

When should you roll your own durable queue versus using a dedicated broker? Here's a comparison:

When to choose what:

• If you need a simple, reliable queue for a small service running on one machine, SQLite is perfect.
• If you're building a distributed system with many producers/consumers, RabbitMQ or Kafka are better — they handle network partitions, replication, and scalability.
• If you already use Redis heavily and your durability requirements are lenient (e.g., cached data, non-critical notifications), Redis with AOF might be enough. But beware: Redis can lose up to 1 second of data depending on your appendfsync policy.

Key differences to remember:

• Durability vs. performance: More durable, more sync writes, higher latency.
• Ack mechanism: SQLite-based queues require manual implementation; brokers have built-in acknowledgment protocols.
• Recovery: SQLite queues need a recover() call; brokers automatically recover and redeliver unacked messages.

Troubleshooting & edge cases

Durable queues are powerful, but they bring their own pitfalls. Here are common ones and how to fix them:

• Messages are lost even with a durable queue. Check your acknowledgment logic. If your consumer acks before safely storing the result, a crash later can still lose data. Follow the rule: ack after processing, not before.
• Duplicate messages after recovery. This is expected with at-least-once delivery. Make your consumer idempotent — use a database unique key or a deduplication layer.
• SQLite is slow under high concurrency. SQLite locks the entire database on writes. For high write rates, use a dedicated broker like Kafka, or batch your writes.
• A crash between enqueue and commit. If commit fails, the message is never persisted. Wrap your enqueue in a transaction and use with conn: for automatic rollback on error.
• In-flight messages get stuck after a crash if you don't call recover(). Ensure your startup sequence always resets in_flight to pending.
• The database file gets corrupted on power loss. SQLite is robust, but if you're on a flaky system, consider enabling WAL mode and using PRAGMA synchronous=FULL for higher safety.

Common Mistakes

• Storing messages only in memory — a crash wipes them permanently. Even if you have a database elsewhere, the queue itself must be durable.
• Acknowledging a message before the consumer finishes writing its output — if a crash happens between the ack and the output write, the message is lost forever.
• Forgetting to reset in_flight messages to pending during recovery — messages that were being processed at crash time will never be retried, causing silent data loss.
• Configuring a broker with low durability settings (e.g., RabbitMQ non-persistent, Kafka acks=1) and assuming you're fully safe — you need persistent / acks=all for true durability.

Variations

• Use a database transaction for enqueue + dequeue — if the same database that stores business data also stores the queue, you can atomically update both, giving you transactional outbox patterns.
• Use a filesystem-based queue — some tools, like sled or custom JSONL files, offer durability with lower overhead than a full DB.
• Use a managed cloud queue — services like AWS SQS or Google Pub/Sub deliver durability out of the box with automatic redelivery and retries, saving you operational effort.

What you learned & what's next

You now understand why durable queues are essential for crash safety: without them, a single crash can lose critical messages. You learned a mental model — the logbook with a whiteboard — and the 5-step process to implement durability: persist before ack, track state, deliver with ack contract, clear on ack, recover after crash. You built a real SQLite-backed durable queue in Python that survives crashes and redelivers in-flight messages. You also compared DIY vs. broker options and handled common edge cases.

You've now mastered crash safety with durable queues. The next step in your Messaging & queues track is to explore dead-letter queues — what happens when a message cannot be processed even after retries? You'll learn how to route poison messages to a separate queue for inspection and failure handling.

Key takeaway: Durable queues are non-negotiable for any system that cannot afford message loss. Always design for at-least-once delivery and make consumers idempotent.

Real-World Use Cases

• Order processing: A customer places an order; the queue persists the order event. Even if the order service crashes mid-process, the order is redelivered on restart, so no orders are lost.
• Email/notification sending: An email service pushes every notification to a durable queue. If the worker crashes after sending the email but before ack, the notification is sent again — users might get a duplicate, but never miss an important alert.
• Financial transactions: A banking app uses a durable queue for transaction processing. If the system crashes before deducting funds, the transaction is retried, ensuring no money is silently lost.

Key Takeaways

• Durable queues persist messages before acknowledging producer receipt, preventing loss on crash.
• Use a state machine (pending, in_flight, done) to track messages and recover them after a crash.
• At-least-once delivery is the standard guarantee; design consumers to be idempotent.
• In-memory queues are fine for volatile data but never for critical messages.
• SQLite is a great lightweight option for simple durable queues; use Kafka/RabbitMQ for large-scale needs.
• Always ack after successful processing, and reset in-flight messages during recovery.

Practice Recap

Now try building your own mini-task queue that survives a simulated crash. Extend the SQLite example to include a process_message function that randomly crashes (by raising an exception) before acking. Restart your queue and confirm the message is redelivered. Then, add a duplicate check to make your consumer idempotent — store processed message IDs in a separate table and skip duplicates. This hands-on exercise will cement the concepts of durability, recovery, and idempotency.