How to Mock Redis Streams Consumer Groups in Python

Simulate Redis Streams producer and consumer group behavior in Python using a standalone mock class for testing and development.

Medium Python 3.9+ Aug 9, 2026 Caching & Redis 14 views 0 copies

Python code

66 lines
Python 3.9+
import time
import json
from collections import defaultdict

class RedisStreamMock:
    def __init__(self):
        self.streams = defaultdict(list)
        self.consumer_groups = defaultdict(dict)
        self.pending_entries = defaultdict(list)

    def xadd(self, stream, fields):
        entry_id = f"{time.time_ns()}-{len(self.streams[stream])}"
        entry = {**fields, "_id": entry_id}
        self.streams[stream].append(entry)
        return entry_id

    def xgroup_create(self, stream, group, start="0"):
        self.consumer_groups[stream][group] = {"last_id": start, "consumers": set()}

    def xreadgroup(self, group, consumer, stream, count=1):
        group_data = self.consumer_groups[stream][group]
        group_data["consumers"].add(consumer)
        last_id = group_data["last_id"]
        available = [e for e in self.streams[stream] if e["_id"] > last_id]
        if not available:
            return []
        selected = available[:count]
        group_data["last_id"] = selected[-1]["_id"]
        for entry in selected:
            self.pending_entries[stream].append({"group": group, "consumer": consumer, "entry": entry})
        return [{"stream": stream, "entries": selected}]

    def xack(self, stream, group, entry_id):
        self.pending_entries[stream] = [
            p for p in self.pending_entries[stream]
            if not (p["group"] == group and p["entry"]["_id"] == entry_id)
        ]
        return True


if __name__ == "__main__":
    redis = RedisStreamMock()

    # Producer adds two messages
    redis.xadd("orders", {"user": "alice", "amount": 100})
    redis.xadd("orders", {"user": "bob", "amount": 250})

    # Create a consumer group
    redis.xgroup_create("orders", "payments")

    # Consumer reads the first message
    result = redis.xreadgroup("payments", "worker-1", "orders", count=1)
    first_entry = result[0]["entries"][0]
    print("Read:", first_entry["user"], first_entry["amount"], first_entry["_id"])

    # Acknowledge it
    redis.xack("orders", "payments", first_entry["_id"])

    # Second consumer reads the next message
    result2 = redis.xreadgroup("payments", "worker-2", "orders", count=1)
    second_entry = result2[0]["entries"][0]
    print("Read:", second_entry["user"], second_entry["amount"], second_entry["_id"])

    # No more messages
    empty = redis.xreadgroup("payments", "worker-2", "orders", count=1)
    print("Empty read:", empty == [])

Output

stdout
Read: alice 100 1699999999999999999-0
Read: bob 250 1699999999999999999-1
Empty read: True

How it works

The mock mimics Redis Streams commands (XADD, XGROUP CREATE, XREADGROUP, XACK) using Python's standard library. Each entry gets a unique ID based on nanosecond time plus a sequence number. Consumer groups track a last-read ID and a set of consumer names, so multiple consumers can read in order. The pending entries list (similar to Redis PEL) is updated when messages are read and cleared upon acknowledgment. This lets you test producer-consumer logic without spinning up a real Redis server.

Common mistakes

  • Assuming message ordering is always preserved—using a simple list here guarantees order, but real Redis may have different guarantees under cluster mode.
  • Forgetting to add a consumer to the group's consumer set on first read—already handled in the mock, but easy to miss when adapting.
  • Not clearing pending entries on acknowledge in your own mock—can lead to memory leaks in tests.

Variations

  1. Use the `fakeredis` pip package for a more feature-complete Redis mock with real command names.
  2. Wrap the mock in a context manager to auto-clean state between tests.

Real-world use cases

  • Testing a payment processing worker that consumes order events from Redis Streams without requiring a Redis instance in CI.
  • Developing a serverless function that reads from a stream and needs local simulation for fast iteration.
  • Writing unit tests for a message queue abstraction that supports multiple consumer groups reading independently.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Caching & Redis

Related tutorials and quizzes for this topic.