Redis Streams Persistence

You've built event-driven systems that work — until the process crashes mid-message. When your in-memory queue loses messages on restart, your consumers miss critical events, and your team loses trust in the pipeline. Redis Streams solve this by giving you a durable, append-only log that survives restarts, supports consumer groups, and lets you replay history on demand. In this lesson, you'll learn how to use Redis Streams for message persistence, from mental model to hands-on code, so your messaging layer finally matches the reliability of your database layer.

The Problem: Lost Messages Break Distributed Systems

Imagine you're processing payment webhooks or order events. Your producer publishes a message to a queue, but before a consumer processes it, the service restarts. With an in-memory queue, that message is gone forever. The payment might be double-charged, an order might never ship, and debugging becomes an archaeology dig through logs.

The pain is real:

• At-least-once delivery is the minimum you need for most business events — every message must be processed at least once.
• Order matters for many workflows (e.g., state transitions), and a single lost message breaks the sequence.
• Replayability is crucial for testing, recovery, or feeding new analytics pipelines — you want to rewind history, not retype it.

Redis Streams directly address this by providing message persistence — data lives on disk and in memory, surviving restarts and crashes.

Core Concept / Mental Model

Think of Redis Streams as a durable, append-only log — like a database table that only allows INSERT, never UPDATE or DELETE (well, almost never). Each entry gets a unique ID and a list of field-value pairs. Consumers read entries in order, but they don't remove them; they track their own cursor (the last-read ID). This is fundamentally different from a queue where messages are removed once consumed.

Key terms to internalize:

• Stream key — the Redis key holding the log (e.g., orders:events).
• Entry — a message with an auto-generated ID like 1695400000000-0 (timestamp-sequence) and fields like {event: 'order.created', order_id: 5}.
• Consumer group — a group of consumers that share the workload; each message is delivered to one consumer in the group.
• Pending Entries List (PEL) — per-consumer record of messages delivered but not yet acknowledged.
• ACK — the consumer's confirmation that a message was fully processed; until ACKed, the message is considered pending.

Analogy: A stream is like a shared team journal. Anyone can write entries. Everyone reads from the same journal, but they maintain a bookmark to their last read position. If someone forgets to mark a page as done, that page remains "pending" for them — no entry is ever erased.

How It Works Step by Step

Let's walk through the typical lifecycle of a Redis Stream message:

1. Producer appends an entry to the stream using XADD. Redis assigns a unique ID and persists the entry (depending on configuration).
2. Consumer group reads new entries using XREADGROUP. Redis fans out each message to exactly one consumer in the group.
3. Consumer processes the message (e.g., writes to a database, sends an email).
4. Consumer ACKs with XACK. This removes the message from the group's PEL.
5. If the consumer crashes before ACKing, the message stays pending — another consumer can claim and reprocess it, ensuring at-least-once delivery.
6. You can read history at any time with XRANGE or XREVRANGE, even after all consumers have processed and ACKed.

Persistence configuration

Redis persistence is a two-layer story:

• RDB snapshots — periodic full backups of the dataset.
• AOF (Append Only File) — logs every write operation for point-in-time recovery.

For streams, enable AOF with fsync policy everysec (default) to limit data loss to at most one second. For zero-loss, use always, but that's slower. In production, you'll also consider Redis replication for high availability.

Hands-On Walkthrough

Let's get our hands dirty. First, make sure Redis is running (version 5.0+ for streams) and install the redis Python client:

pip install redis

1. Producing messages with persistence

import redis

r = redis.Redis(host='localhost', port=6379, decode_responses=True)

# Append a message to the stream
stream_key = "events:orders"
message = {"event": "order.created", "order_id": "1234", "amount": "99.50"}
message_id = r.xadd(stream_key, message)
print(f"Message ID: {message_id}")

# Output: Message ID: 1695400000000-0

Run the producer, then restart Redis, and the message will still be there — persistence at work.

2. Reading with consumer groups (the durable pattern)

import redis
import time

r = redis.Redis(host='localhost', port=6379, decode_responses=True)
stream_key = "events:orders"
group_name = "order-processors"
consumer_name = "worker-1"

try:
    r.xgroup_create(stream_key, group_name, id='0')
except redis.exceptions.ResponseError as e:
    # Group already exists
    if "BUSYGROUP" not in str(e):
        raise

while True:
    # Read up to 5 new messages for this consumer, blocking for 2 seconds
    entries = r.xreadgroup(group_name, consumer_name, {stream_key: '>'}, count=5, block=2000)
    if not entries:
        continue
    for stream, messages in entries:
        for msg_id, fields in messages:
            print(f"Processing message {msg_id}: {fields}")
            # Simulate work
            time.sleep(0.1)
            # Acknowledge to remove from PEL
            r.xack(stream_key, group_name, msg_id)

If the consumer crashes inside time.sleep(0.1) before ACK, the message remains pending and can be recovered.

3. Replaying and recovering pending messages

# Read history from the beginning (id='0')
historic = r.xrange(stream_key, min='-', max='+')
print("Historic messages:")
for msg_id, fields in historic:
    print(msg_id, fields)

# Check pending messages for the group
pending = r.xpending(stream_key, group_name)
print(f"Pending: {pending}")

# Claim a message from another consumer (e.g., after crash) and reprocess
claimed = r.xclaim(stream_key, group_name, consumer_name, min_idle_time=60_000, ids=[pending_id])

This pattern is the backbone of at-least-once delivery: messages that weren't ACKed get reclaimed and processed again.

Compare Options / When to Choose What

Redis Streams aren't the only persistence game in town. Here's a quick comparison:

When to choose Redis Streams: - You already run Redis and want minimal new infrastructure. - You need message replay and consumer groups without a full Kafka cluster. - Your throughput needs are modest (thousands of messages/sec, not millions).

When to choose something else: - Kafka if you need massive scale, multiple consumers per partition, or long-term retention (weeks/months). - RabbitMQ if you need complex routing, dead-letter exchanges, or the classic queue semantics.

Troubleshooting & Edge Cases

Even with persistence, things can go sideways. Here are common pitfalls and their fixes:

1. Messages lost on restart despite AOF enabled

• Symptom: Messages vanish after restart.
• Cause: AOF is disabled, or fsync is set to no/everysec (default tolerates ≤1s loss).
• Fix: Enable AOF in redis.conf: bash appendonly yes appendfsync everysec # or always for stricter Then restart Redis. Test by killing Redis and checking the stream.

2. Consumer group reads duplicate messages after crash

• Symptom: A message is processed twice.
• Cause: At-least-once delivery — the consumer crashed before ACK.
• Fix: Make consumers idempotent (e.g., check if order_id was already processed) to safely handle duplicates. Read your consumer's XACK logic — ensure ACK happens after side effects commit.

3. Consumer group blocks forever when no messages

• Symptom: XREADGROUP hangs.
• Cause: Blocking call without timeout or with count not reached.
• Fix: Use block=2000 to poll, or block=0 for indefinite blocking with a separate timeout on the client side (e.g., socket_timeout).

4. Stream length grows unbounded

• Symptom: Redis memory or disk fills up.
• Cause: Streams are append-only — entries never removed automatically.
• Fix: Trim with XTRIM or MAXLEN on XADD: python r.xadd(stream_key, message, maxlen=10000) or explicit r.xtrim(stream_key, maxlen=5000) for a retention policy.

5. Consumer group auto-creation error

• Symptom: NOGROUP No such consumer group.
• Cause: Calling XREADGROUP before creating the group.
• Fix: Always call XGROUP CREATE first (you did in the walkthrough).

What You Learned & What's Next

You now understand how to use Redis Streams for message persistence: the append-only log model, producer/consumer groups with ACK, replay and recovery, and when to choose streams over alternatives. You completed a hands-on exercise proving that messages survive restarts and that pending-message recovery works.

Next, you'll explore dead-letter queues and error handling — how to isolate messages that keep failing and alert operators instead of blocking the pipeline. With Redis Streams persistence under your belt, you're ready to build resilient event pipelines that never lose a message (well, almost never).