Redis Acknowledge & Trim Streams

Streams are one of Redis's most powerful data structures—but without proper acknowledgment and trimming, they can quickly become a memory drain and a source of message replay nightmares. This lesson teaches you how to clean up after yourself using XACK to confirm message consumption and XTRIM to cap stream length, keeping your streams lean, efficient, and production-ready.

The problem this lesson solves

Imagine you're building a real-time event pipeline—order updates, user activity, or IoT sensor readings—using Redis Streams. Your consumers read new messages, process them, and move on. But what happens when a consumer crashes midway? Without a way to mark messages as "done," you risk either replaying processed events (duplicate side effects) or losing unprocessed events forever. Worse, streams grow unbounded: a high-throughput stream can balloon to gigabytes, exhausting memory and degrading performance.

This lesson solves two concrete problems:

• Message acknowledgment: Letting Redis know which messages a consumer group has successfully processed, so they won't be redelivered.
• Stream trimming: Removing old messages from a stream to control memory usage, while preserving enough history for late-arriving consumers.

By the end, you'll treat stream management as a first-class operation, not an afterthought.

Core concept / mental model

Think of a Redis Stream as a persistent log—like a publish‑subscribe channel but with a durable, ordered message history. Each consumer group acts like a bookmark that tracks which messages its members have read. But reading a message does not automatically mean it's processed.

• XACK is the "I've handled this message" signal. Without it, the group's pending entries list (PEL) keeps the message, and another consumer may claim it if the first one fails.
• XTRIM is the memory cap. It lets you bound the stream to a maximum length (e.g., 10,000 entries) by evicting the oldest messages. Crucially, XTRIM does not delete pending messages—they remain in the PEL even if the stream entry is gone, which can cause dead-letter scenarios.

Diagram in words:

Stream: [msg1] [msg2] [msg3] [msg4] ... [msg1000]
        ↑                                ↑
      Consumer reads msg2              XTRIM cuts here
      → XACK msg2                     (oldest entries removed)

Without XACK, the PEL grows unbounded, and consumers may re‑read stale messages. Without XTRIM, the stream itself grows without bound.

How it works step by step

Acknowledgment and trimming happen in distinct but complementary workflows:

1. Consumer group setup

Before you can acknowledge messages, you must have a consumer group created with XGROUP CREATE.

2. Reading messages

Consumers read messages using XREADGROUP. Redis returns only messages that haven't been delivered yet (unless you use the > special ID).

3. Processing logic

Your application processes the message—e.g., sends an email, updates a database, triggers a webhook.

4. Acknowledging with XACK

After successful processing, call XACK <stream> <group> <message-id> to remove the message from the group's PEL. Now Redis knows the message is handled.

5. Trimming with XTRIM

Periodically (or after each batch), use XTRIM <stream> MAXLEN [~] <count> to evict the oldest entries. The ~ flag (tilde) allows Redis to perform a more efficient, lazy eviction—good for production.

6. Monitoring pending entries

Use XPENDING <stream> <group> to see which messages remain unacknowledged. This is critical for detecting stuck consumers or messages that need manual reprocessing.

Hands-on walkthrough

Let's put it all together. We'll create a stream for order events, add a few messages, read them with a consumer group, acknowledge them, and then trim the stream.

Prerequisites

Make sure Redis (7.0+) is running locally. You'll use redis-cli and optionally a Python script with the redis-py library.

1. Create a stream and consumer group

# Add an event to the stream (creates the stream if absent)
XADD orders * event new_order user_id 42 amount 29.99
XADD orders * event payment_complete user_id 42 txn_id abc123

# Create a consumer group
XGROUP CREATE orders order-processors $ MKSTREAM

$ means "start reading from the newest message." MKSTREAM creates the stream if it doesn't exist.

2. Read messages with the group

# Consumer 'worker-1' reads the next unread message
XREADGROUP GROUP order-processors worker-1 COUNT 1 STREAMS orders >

Expected output (IDs will differ):

1) 1) "orders"
   2) 1) 1) "1700000000000-0"
         2) 1) "event"
            2) "new_order"
            3) "user_id"
            4) "42"
            5) "amount"
            6) "29.99"

3. Acknowledge the message

Once the message is processed, acknowledge it:

XACK orders order-processors 1700000000000-0

Expected output: (integer) 1 — one message acknowledged.

4. Check pending entries

XPENDING orders order-processors

Expected output: 1) (integer) 0 (if you acknowledged the only message).

5. Add more messages and trim the stream

# Add 100 events quickly (simulated in CLI)
for i in $(seq 1 100); do
  redis-cli XADD orders * event heartbeat seq $i
done

# Trim to the last 50 entries
XTRIM orders MAXLEN 50

Expected output: (integer) 52 — Redis removed 52 old entries (the first new_order and payment_complete plus the first 50 heartbeats, keeping the newest 50).

6. Python equivalent using redis-py

import redis

r = redis.Redis(decode_responses=True)

# Add messages
r.xadd('orders', {'event': 'new_order', 'user_id': '42'})
r.xadd('orders', {'event': 'payment_complete', 'user_id': '42'})

# Create group
r.xgroup_create('orders', 'order-processors', id='$', mkstream=True)

# Read
results = r.xreadgroup('order-processors', 'worker-1', {'orders': '>'}, count=1)
for stream, messages in results:
    for msg_id, msg_data in messages:
        print(f"Processing {msg_id}: {msg_data}")
        # Simulate processing
        r.xack('orders', 'order-processors', msg_id)

# Trim
r.xtrim('orders', maxlen=10)

Expected output:

Processing 1700000000000-0: {'event': 'new_order', 'user_id': '42'}

Pro tip: Always XACK as close to the end of processing as possible. If you acknowledge before the real work (e.g., before a database write), a crash may cause data loss. If you acknowledge too late, other consumers may claim and re-process.

Compare options / when to choose what

Key takeaway: XACK is about consumption state; XTRIM is about stream size. You need both.

Troubleshooting & edge cases

1. "I acknowledged a message, but it still appears in XPENDING"

• Cause: You're checking the wrong consumer group or stream name (case-sensitive in Redis).
• Fix: Double-check group and stream names. Use XINFO STREAM orders to list groups.

2. "XACK returns 0 (integer)"

• Cause: The message ID you provided is not in the group's PEL. Maybe it was already acknowledged, or it belongs to a different group.
• Fix: Confirm the message ID with XPENDING orders order-processors. Ensure you're using the exact ID (including timestamp and sequence).

3. "I trimmed the stream, but pending messages disappeared!"

• Cause: XTRIM deleted the underlying stream entries, but the pending entries in the PEL remain. They become orphaned—Redis tracks them but the data is gone.
• Fix: Never trim a stream that has unacknowledged messages you care about. Either acknowledge first or use a larger trim limit. Check XPENDING before trimming.

4. "My stream is still huge after XTRIM MAXLEN 10000"

• Cause: MAXLEN without the tilde (~) is exact but slow; with ~ it's approximate but faster. Also, XTRIM removes only enough to reach the target—if the stream is smaller, nothing happens.
• Fix: Use XTRIM orders MAXLEN ~ 10000 for production. Verify with XLEN orders.

5. "A consumer group claims a message, but the original consumer is still working on it"

• Cause: Another consumer used XCLAIM to take over a pending message that was idle for too long.
• Fix: Increase the idle timeout in XCLAIM or use XAUTOCLAIM with a sensible limit. Never auto-claim without ensuring the original consumer is truly dead.

What you learned & what's next

You now understand the critical difference between reading a message and acknowledging it. You've seen how XACK keeps your consumer groups clean and how XTRIM keeps your streams memory‑efficient. You've also discovered common pitfalls like orphaned pending entries and incorrect group names.

Learning objectives achieved:

✅ Explain the core idea behind XACK (per‑message acknowledgment) and XTRIM (stream length bounding). ✅ Complete a practical exercise using both CLI and Python.

What's next: Consumer groups often need to handle failures gracefully. The next lesson, Claiming and Auto‑claiming Messages, teaches you how to reassign unprocessed messages from dead consumers, ensuring no event is ever lost.