How to Mock Kafka Topic Partitions with a Python dict of lists

Mocks a Kafka topic and its partitions using a defaultdict of lists to simulate message production, consumption, and per-partition counts.

Easy Python 3.9+ Aug 9, 2026 Streaming & messaging 15 views 0 copies

Python code

36 lines
Python 3.9+
from collections import defaultdict

class KafkaTopicPartitionMock:
    """A simple mock for Kafka topic-partition assignment using dict of lists."""

    def __init__(self, topic):
        self.topic = topic
        self.partitions = defaultdict(list)  # partition_id -> list of messages

    def produce(self, message, partition=None):
        """Assign message to a partition. If None, use round-robin."""
        if partition is None:
            partition = len(self.partitions) % 3  # mock 3 partitions
        self.partitions[partition].append(message)

    def consume(self, partition):
        """Retrieve all messages from a partition."""
        return self.partitions.get(partition, [])

    def show_partition_counts(self):
        """Display message counts per partition."""
        return {p: len(msgs) for p, msgs in sorted(self.partitions.items())}


if __name__ == "__main__":
    topic = KafkaTopicPartitionMock("orders")
    topic.produce("order-1", partition=0)
    topic.produce("order-2", partition=1)
    topic.produce("order-3", partition=2)
    topic.produce("order-4", partition=0)
    topic.produce("order-5", partition=1)

    print("Partition counts:", topic.show_partition_counts())
    print("Partition 0:", topic.consume(0))
    print("Partition 1:", topic.consume(1))
    print("Partition 2:", topic.consume(2))

Output

stdout
Partition counts: {0: 2, 1: 2, 2: 1}
Partition 0: ['order-1', 'order-4']
Partition 1: ['order-2', 'order-5']
Partition 2: ['order-3']

How it works

The defaultdict(list) automatically creates a new list when a partition key is first accessed, simplifying partition creation. produce appends messages to the selected partition's list, while consume retrieves all messages for that partition with .get() to avoid KeyError for missing partitions. show_partition_counts uses a sorted dict comprehension to return a readable per-partition message count. This mimics the basic behavior of Kafka's partition storage while remaining lightweight and dependency-free.

Common mistakes

  • Using a regular dict without default values, causing KeyError on first produce to a new partition
  • Forgetting that defaultdict passes the factory to all missing keys, including when checking `in` or `.get()`
  • Not sorting partitions when displaying counts, leading to non-deterministic output order
  • Assuming `consume` removes messages; in Kafka, consumption does not delete the partition data

Variations

  1. Use a `dict[int, deque]` to simulate a queue with fast popleft for consumer groups
  2. Implement round-robin partition assignment by tracking a counter instead of using `len(self.partitions) % 3`

Real-world use cases

  • Unit testing a Kafka consumer/producer pipeline without needing a real broker in CI
  • Simulating partition order and message distribution for offline analysis of streaming logic
  • Prototyping partition key strategies in a local script before deploying to production Kafka

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Streaming & messaging

Related tutorials and quizzes for this topic.