Redis Streams Basics

Ever tried to build a real-time event pipeline or message queue and found yourself wrestling with complex messaging systems, only to need something simpler and faster? Redis Streams solves this exact pain point — it's a lightweight, append-only log data structure that enables you to produce, consume, and fan-out messages with near-zero latency, all within your familiar Redis ecosystem. This lesson will transform you from a Redis beginner into someone who can build robust, production-grade streaming workflows.

The problem this lesson solves

Traditional message brokers like Kafka, RabbitMQ, or NATS are powerful but often overkill for microservices, real-time dashboards, or distributed task queues. They require separate infrastructure, complex configuration, and a steep learning curve. Meanwhile, simpler solutions like Redis Pub/Sub lack message persistence — if a consumer disconnects, those messages are lost forever. Redis Streams bridges this gap by providing an append‑only log that persists messages, supports multiple consumer groups, and allows at-least-once delivery semantics — all using a single, well‑known database that you likely already have running.

Core concept / mental model

Think of a Redis Stream as a never-ending log file stored in Redis. Each entry is a message with a unique auto‑generated ID (in the format <timestamp>-<sequenceNumber>) and a set of field‑value pairs (like a hash). Multiple producers can append messages to the stream, and multiple consumers can read messages starting from any point. You can even create consumer groups — named sets of consumers that cooperate to process messages from the same stream, ensuring each message is processed only once by the group.

Key definitions

• Stream: A Redis key of type STREAM that stores ordered messages.
• Entry: A single message in the stream — uniquely identified by its ID.
• Consumer Group: A named group of one or more consumers that coordinate to process messages from the same stream.
• Consumer: A member of a consumer group that reads and acknowledges messages.
• Pending Entry List (PEL): A per-consumer list of messages that have been delivered but not yet acknowledged.

Pro tip 💡 Redis Streams are not like Kafka partitions — there’s no built‑in partitioning mechanism. All messages for a stream live in a single Redis node (sharding is manual). For horizontal scaling, use multiple streams sharded by a key or use Redis Cluster.

How it works step by step

1. Creating a Stream

Streams are created implicitly when you add the first message using XADD. The stream automatically grows as new entries are appended. Unlike lists, you cannot manipulate arbitrary positions; you always append new messages to the end.

2. Adding Messages with XADD

XADD is the main write command:

XADD <streamKey> [MAXLEN ~ <count>] [*|<id>] <field1> <value1> [<field2> <value2> ...]

• The * tells Redis to auto‑generate a sequential ID based on the server's clock.
• MAXLEN ~ <count> trims the stream to approximately <count> entries (non‑blocking).
• You can also supply a custom ID (e.g., 0-0), but this is rare — * is safest.

3. Reading Messages with XREAD

XREAD reads messages from one or more streams, optionally starting from a given ID:

XREAD COUNT <count> [BLOCK <milliseconds>] STREAMS <key1> <key2> ... <id1> <id2> ...

• COUNT limits the number of returned entries.
• BLOCK enables blocking read (like pub/sub’s SUBSCRIBE) — waits for new data for the specified milliseconds (or indefinitely with 0).
• You can pass $ to read only new messages (from the tail), or a specific ID to read from that point forward.

4. Consumer Groups: XGROUP, XREADGROUP, XACK

This is where Streams truly shine compared to Pub/Sub or Lists: 1. Create group: XGROUP CREATE <streamKey> <groupName> <id> 2. Consume: XREADGROUP GROUP <groupName> <consumerName> [COUNT <n>] [BLOCK <ms>] STREAMS <streamKey> > - The > special ID tells Redis to return only messages not yet delivered to any consumer in that group. - Each consumer gets a unique subset of messages — no duplicates across consumers. 3. Acknowledge: XACK <streamKey> <groupName> <id1> <id2> ...

5. Acknowledging Messages

After processing a message, the consumer must XACK it to remove it from the PEL. If you don’t ack, the message stays pending and can be claimed by another consumer (failure recovery).

Hands-on walkthrough

Let’s build a simple event pipeline: a producer adds user‑action logs to a stream, and two competing consumers process them concurrently.

Setup: Start Redis (if not already)

redis-server --port 6379

Step 1: Add some messages

redis-cli XADD user_actions * user_id 42 action login
redis-cli XADD user_actions * user_id 73 action signup
redis-cli XADD user_actions * user_id 15 action logout

Expected output (auto‑generated IDs):

1722335678901-0
1722335678902-0
1722335678903-0

Step 2: Read from the start

redis-cli XRANGE user_actions - +

Expected output (abbreviated):

1) 1) "1722335678901-0"
   2) 1) "user_id"
      2) "42"
      3) "action"
      4) "login"
...

Step 3: Create a consumer group

redis-cli XGROUP CREATE user_actions my_group 0-0

This creates a group named my_group that will start reading all existing messages (0-0 means from the beginning). Use $ to start from new messages only.

Step 4: Consume with a consumer

redis-cli XREADGROUP GROUP my_group consumer1 COUNT 2 BLOCK 2000 STREAMS user_actions >

Expected output:

1) 1) "user_actions"
   2) 1) 1) "1722335678901-0"
         2) 1) "user_id"
            2) "42"
            3) "action"
            4) "login"
      2) 1) "1722335678902-0"
         2) 1) "user_id"
            2) "73"
            3) "action"
            4) "signup"

Step 5: Acknowledge processed messages

redis-cli XACK user_actions my_group 1722335678901-0 1722335678902-0

Now those two messages are no longer pending.

Python Example

Here’s the same flow in Python using redis-py:

import redis

r = redis.Redis(host='localhost', port=6379, decode_responses=True)

# Add messages
r.xadd('user_actions', {'user_id': '42', 'action': 'login'})
r.xadd('user_actions', {'user_id': '73', 'action': 'signup'})
r.xadd('user_actions', {'user_id': '15', 'action': 'logout'})

# Create group (idempotent — use mkstream=True if stream might not exist)
r.xgroup_create('user_actions', 'my_group', id='0-0', mkstream=True)

# Consume 2 messages
results = r.xreadgroup('my_group', 'consumer1', {'user_actions': '>'}, count=2)
for stream_key, entries in results:
    for msg_id, fields in entries:
        print(f"Processing {msg_id}: {fields}")
        # Acknowledge after processing
        r.xack('user_actions', 'my_group', msg_id)

Expected output:

Processing 1722335678901-0: {'user_id': '42', 'action': 'login'}
Processing 1722335678902-0: {'user_id': '73', 'action': 'signup'}

Compare options / when to choose what

Choose Redis Streams when: You need persistent message queues, support for consumer groups, and at‑least‑once delivery without adding Kafka or RabbitMQ. Ideal for microservices, real‑time analytics pipelines, or distributed task queues.

Troubleshooting & edge cases

Stream trimming with MAXLEN

redis-cli XADD user_actions MAXLEN ~ 1000 * user_id 99 action visit

~ means “approximate” — Redis may trim slightly more or fewer entries for performance. Use = for exact (but slower) trimming.

Consumer group not found

If you get NOGROUP error, use XGROUP CREATE ... MKSTREAM or create the stream first with XADD.

PEL growing unexpectedly

If consumers crash without acknowledging, the PEL grows. Use XPENDING to inspect pending messages and XCLAIM to reassign them to another consumer for reprocessing.

Duplicate processing in groups

If you don’t XACK after processing, and the consumer reconnects, XREADGROUP may return the same message again. Always XACK in a finally block or after successful processing.

Wrong ID format when reading

When using XREAD with a custom ID, ensure it’s in the <timestamp>-<seq> format (e.g., 1722335678901-0). Use 0-0 for the start, or $ for the latest.

Performance considerations

• Streams with millions of entries consume memory. Trim old messages using XTRIM or set MAXLEN on XADD.
• Blocking reads (BLOCK option) hold a client connection open — respect timeouts and reconnect if needed.

What you learned & what's next

You now understand the core concepts of Redis Streams: how to produce, consume, and manage messages with consumer groups, and when Streams outperform Pub/Sub or external message brokers. You practiced the essential commands (XADD, XREAD, XGROUP, XREADGROUP, XACK) and can apply them in Python applications.

In the next lesson, you'll dive deeper into consumer failure handling — using XCLAIM and XPENDING for robust worker recovery, and exploring XAUTOCLAIM for automatic rebalancing. You'll also learn how to monitor stream lag and implement dead‑letter queues.