Use Consumer Groups with Streams

Imagine a single worker frantically polling a message queue while messages pile up during a traffic spike — exactly what breaks a system. Consumer groups in Redis Streams solve this by letting multiple consumers cooperatively process messages without duplication, providing fault tolerance and scalability. Think of it as a team conveyor belt: each worker picks one item, processes it, and marks it done; if a worker drops out, its pending items are reassigned.

The problem this lesson solves

Standard Redis Streams consumers read independently: every consumer gets every message. That’s fine for broadcast (e.g., sending alerts to all services), but disastrous for task queues where each job must be processed exactly once. Without consumer groups:

• Duplicate processing — multiple consumers grab the same message.
• No recovery — crashed consumers lose their in-flight messages.
• No auto-balancing — fast consumers idle while slow ones queue up.

Consumer groups introduce coordinated consumption: each message goes to one consumer within the group. Redis tracks which messages each consumer has seen, which are pending, and which are acknowledged. This pattern is critical for order processing, event logging pipelines, and background job systems.

Core concept / mental model

Picture a bus stop where passengers (messages) arrive continuously. Without a dispatcher, every bus picks up every passenger — chaos. A consumer group is the dispatcher that assigns each passenger to exactly one bus driver (consumer). The dispatcher also records which passengers are still waiting (pending), which boarded (acknowledged), and which bus crashed so another can take its pending passengers.

Key components:

• Stream — the message log (e.g., orders:stream).
• Consumer group — logical name (e.g., order-processors).
• Consumers — named workers within the group (e.g., worker-1, worker-2).
• Pending Entries List (PEL) — per-consumer list of unacknowledged messages.
• Last Delivered ID — pointer to the last message handed out.

Messages are read once per group using XREADGROUP, and acknowledged with XACK after successful processing. Messages stay in the stream indefinitely (or until trimmed); acknowledged messages are simply removed from each consumer's PEL.

How consumer groups differ from pub/sub

Unlike Pub/Sub (fire-and-forget), consumer groups:

How it works step by step

1. Create a stream and group

You must create the stream first (even with a dummy message) or use XGROUP CREATE with the MKSTREAM option.

XGROUP CREATE orders:stream order-processors $ MKSTREAM

• orders:stream — stream key (created automatically if MKSTREAM is used).
• order-processors — group name (unique per stream).
• $ — start reading only new messages after creation. Use 0 to read all historical messages.

2. Consumers join and read

Each consumer independently calls XREADGROUP with its unique name.

# Consumer worker-1 reads one pending message, waiting up to 10 seconds
XREADGROUP GROUP order-processors worker-1 BLOCK 10000 COUNT 1 STREAMS orders:stream >

The > special ID tells Redis: give me the next unassigned message for this group. Redis assigns it exclusively to worker-1 and adds it to worker-1's PEL.

3. Process and acknowledge

After processing, the consumer calls XACK to remove the message from its PEL.

XACK orders:stream order-processors 1720000000000-0

Where 1720000000000-0 is the message ID returned from XREADGROUP. If you don't acknowledge, the message stays in the PEL and counts as pending.

4. Claim unprocessed messages (failure recovery)

If a consumer crashes, its pending messages are stuck. Another consumer can claim them after a timeout (MIN-IDLE-TIME in milliseconds).

XCLAIM orders:stream order-processors worker-2 30000 1720000000000-0

This transfers 1720000000000-0 from worker-1 (inactive >30s) to worker-2. The worker-2 then processes and acknowledges.

Hands-on walkthrough

Let's simulate a real scenario: a job queue with two workers.

Setup: create stream and add messages

redis-cli

XADD jobs:stream * task email to alice@example.com
XADD jobs:stream * task email to bob@example.com
XADD jobs:stream * task report to /tmp/report.csv
XGROUP CREATE jobs:stream job-workers $ MKSTREAM

Worker-1 reads a message

XREADGROUP GROUP job-workers worker-1 COUNT 1 STREAMS jobs:stream >

Expected output:

1) 1) "jobs:stream"
   2) 1) 1) "1720000000001-0"
         2) 1) "task"
            2) "email"
            3) "to"
            4) "alice@example.com"

Worker-1 acknowledges:

XACK jobs:stream job-workers 1720000000001-0

Worker-2 reads the next message

XREADGROUP GROUP job-workers worker-2 COUNT 1 STREAMS jobs:stream >

Expected output:

1) 1) "jobs:stream"
   2) 1) 1) "1720000000002-0"
         2) 1) "task"
            2) "email"
            3) "to"
            4) "bob@example.com"

Notice worker-1 did not see bob's message — load balancing works.

Simulate crash and claim

Worker-1 fails after reading third message. Worker-2 claims it after 30 seconds idle time:

XCLAIM jobs:stream job-workers worker-2 30000 1720000000003-0

Worker-2 now processes and acknowledges:

XACK jobs:stream job-workers 1720000000003-0

Check pending messages for a consumer

XPENDING jobs:stream job-workers - + 10 worker-1

This shows all pending messages for worker-1 (none if acknowledged).

Compare options / when to choose what

Pro tip: Use consumer groups when you need at-least-once delivery and parallel processing. Combine with XINFO GROUPS to monitor group state.

Troubleshooting & edge cases

"NOGROUP No such consumer group"

Forgot to create the group. Run XGROUP CREATE with MKSTREAM if the stream doesn't exist yet.

Consumer reads same message twice

• Consumers in the same group must have unique names.
• If two consumers use the same name, Redis treats them as the same consumer — they won't load balance properly.
• Use hostname + process ID (e.g., worker-a-12345) to guarantee uniqueness.

PEL grows unbounded

Every message delivered but not acknowledged stays in the PEL. If consumers crash frequently without XACK, PEL can consume memory. Mitigation:

• Set a PEL size limit using XGROUP SETID?
• Use XAUTOCLAIM (Redis 6.2+) to automatically claim and purge stale pending messages.

"The ID specified in XADD is equal or smaller than the expected minimum id"

You tried to use a custom ID like 0-0 or a past timestamp. Streams enforce monotonic IDs. Use * for auto-generated, or ensure your IDs are strictly increasing.

Loss of messages on consumer restart

If you restart a consumer and use the same name, its PEL is still intact. Use XREADGROUP without > to read your own historical pending messages:

XREADGROUP GROUP job-workers worker-1 STREAMS jobs:stream 0

This returns all unacknowledged messages previously assigned to worker-1.

What you learned & what's next

You now understand how to use consumer groups with streams:

• Create groups with XGROUP CREATE
• Read messages exclusively with XREADGROUP ... >
• Acknowledge with XACK
• Recover failures with XCLAIM
• Monitor pending entries with XPENDING

Consumer groups unlock reliable, scalable message processing in Redis — essential for job queues, event streaming, and distributed task schedulers.

Next lesson: "Trim Streams & Manage Memory" — learn how to cap your PEL and stream size to prevent memory exhaustion, and use XTRIM efficiently.