How to Mock Redis Streams Consumer Groups in Python
Simulate Redis Streams producer and consumer group behavior in Python using a standalone mock class for testing and development.
Python code
66 linesimport time
import json
from collections import defaultdict
class RedisStreamMock:
def __init__(self):
self.streams = defaultdict(list)
self.consumer_groups = defaultdict(dict)
self.pending_entries = defaultdict(list)
def xadd(self, stream, fields):
entry_id = f"{time.time_ns()}-{len(self.streams[stream])}"
entry = {**fields, "_id": entry_id}
self.streams[stream].append(entry)
return entry_id
def xgroup_create(self, stream, group, start="0"):
self.consumer_groups[stream][group] = {"last_id": start, "consumers": set()}
def xreadgroup(self, group, consumer, stream, count=1):
group_data = self.consumer_groups[stream][group]
group_data["consumers"].add(consumer)
last_id = group_data["last_id"]
available = [e for e in self.streams[stream] if e["_id"] > last_id]
if not available:
return []
selected = available[:count]
group_data["last_id"] = selected[-1]["_id"]
for entry in selected:
self.pending_entries[stream].append({"group": group, "consumer": consumer, "entry": entry})
return [{"stream": stream, "entries": selected}]
def xack(self, stream, group, entry_id):
self.pending_entries[stream] = [
p for p in self.pending_entries[stream]
if not (p["group"] == group and p["entry"]["_id"] == entry_id)
]
return True
if __name__ == "__main__":
redis = RedisStreamMock()
# Producer adds two messages
redis.xadd("orders", {"user": "alice", "amount": 100})
redis.xadd("orders", {"user": "bob", "amount": 250})
# Create a consumer group
redis.xgroup_create("orders", "payments")
# Consumer reads the first message
result = redis.xreadgroup("payments", "worker-1", "orders", count=1)
first_entry = result[0]["entries"][0]
print("Read:", first_entry["user"], first_entry["amount"], first_entry["_id"])
# Acknowledge it
redis.xack("orders", "payments", first_entry["_id"])
# Second consumer reads the next message
result2 = redis.xreadgroup("payments", "worker-2", "orders", count=1)
second_entry = result2[0]["entries"][0]
print("Read:", second_entry["user"], second_entry["amount"], second_entry["_id"])
# No more messages
empty = redis.xreadgroup("payments", "worker-2", "orders", count=1)
print("Empty read:", empty == [])
Output
Read: alice 100 1699999999999999999-0
Read: bob 250 1699999999999999999-1
Empty read: True
How it works
The mock mimics Redis Streams commands (XADD, XGROUP CREATE, XREADGROUP, XACK) using Python's standard library. Each entry gets a unique ID based on nanosecond time plus a sequence number. Consumer groups track a last-read ID and a set of consumer names, so multiple consumers can read in order. The pending entries list (similar to Redis PEL) is updated when messages are read and cleared upon acknowledgment. This lets you test producer-consumer logic without spinning up a real Redis server.
Common mistakes
- Assuming message ordering is always preserved—using a simple list here guarantees order, but real Redis may have different guarantees under cluster mode.
- Forgetting to add a consumer to the group's consumer set on first read—already handled in the mock, but easy to miss when adapting.
- Not clearing pending entries on acknowledge in your own mock—can lead to memory leaks in tests.
Variations
- Use the `fakeredis` pip package for a more feature-complete Redis mock with real command names.
- Wrap the mock in a context manager to auto-clean state between tests.
Real-world use cases
- Testing a payment processing worker that consumes order events from Redis Streams without requiring a Redis instance in CI.
- Developing a serverless function that reads from a stream and needs local simulation for fast iteration.
- Writing unit tests for a message queue abstraction that supports multiple consumer groups reading independently.
Sponsored
More from Caching & Redis
- Cache Asides in Python with a Read-Through Loader easy
- Cache Data in Redis with Python easy
- Cache Penetration Null Object Mock in Python medium
- Cache Stampede Prevention with SingleFlight in Python medium
- Cache Warming with Python: Preload Hot Keys easy
- Coalescing duplicate in-flight requests: one shared result for concurrent callers hard
Keep learning
Related tutorials and quizzes for this topic.