How to Mock Kafka Topic Partitions with a Python dict of lists
Mocks a Kafka topic and its partitions using a defaultdict of lists to simulate message production, consumption, and per-partition counts.
Python code
36 linesfrom collections import defaultdict
class KafkaTopicPartitionMock:
"""A simple mock for Kafka topic-partition assignment using dict of lists."""
def __init__(self, topic):
self.topic = topic
self.partitions = defaultdict(list) # partition_id -> list of messages
def produce(self, message, partition=None):
"""Assign message to a partition. If None, use round-robin."""
if partition is None:
partition = len(self.partitions) % 3 # mock 3 partitions
self.partitions[partition].append(message)
def consume(self, partition):
"""Retrieve all messages from a partition."""
return self.partitions.get(partition, [])
def show_partition_counts(self):
"""Display message counts per partition."""
return {p: len(msgs) for p, msgs in sorted(self.partitions.items())}
if __name__ == "__main__":
topic = KafkaTopicPartitionMock("orders")
topic.produce("order-1", partition=0)
topic.produce("order-2", partition=1)
topic.produce("order-3", partition=2)
topic.produce("order-4", partition=0)
topic.produce("order-5", partition=1)
print("Partition counts:", topic.show_partition_counts())
print("Partition 0:", topic.consume(0))
print("Partition 1:", topic.consume(1))
print("Partition 2:", topic.consume(2))
Output
Partition counts: {0: 2, 1: 2, 2: 1}
Partition 0: ['order-1', 'order-4']
Partition 1: ['order-2', 'order-5']
Partition 2: ['order-3']
How it works
The defaultdict(list) automatically creates a new list when a partition key is first accessed, simplifying partition creation. produce appends messages to the selected partition's list, while consume retrieves all messages for that partition with .get() to avoid KeyError for missing partitions. show_partition_counts uses a sorted dict comprehension to return a readable per-partition message count. This mimics the basic behavior of Kafka's partition storage while remaining lightweight and dependency-free.
Common mistakes
- Using a regular dict without default values, causing KeyError on first produce to a new partition
- Forgetting that defaultdict passes the factory to all missing keys, including when checking `in` or `.get()`
- Not sorting partitions when displaying counts, leading to non-deterministic output order
- Assuming `consume` removes messages; in Kafka, consumption does not delete the partition data
Variations
- Use a `dict[int, deque]` to simulate a queue with fast popleft for consumer groups
- Implement round-robin partition assignment by tracking a counter instead of using `len(self.partitions) % 3`
Real-world use cases
- Unit testing a Kafka consumer/producer pipeline without needing a real broker in CI
- Simulating partition order and message distribution for offline analysis of streaming logic
- Prototyping partition key strategies in a local script before deploying to production Kafka
Sponsored
More from Streaming & messaging
- At Most Once Fire-and-Forget Mock in Python easy
- Batch Consume Process Commit Pattern in Python medium
- Build a Streaming Messaging Helper in Python easy
- Dead Letter Queue Failed Messages List Mock in Python easy
- Dedupe processed message IDs in Python easy
- Event Envelope with Schema Version Field in Python easy
Keep learning
Related tutorials and quizzes for this topic.