Mock Kafka Consumer Group Partition Assignment in Python

Simulates a Kafka consumer group's round-robin partition assignment with a Python class and prints assignments per consumer.

Medium Python 3.9+ Aug 9, 2026 Streaming & messaging 14 views 0 copies

Python code

41 lines
Python 3.9+
from collections import defaultdict


class ConsumerGroupAssignment:
    def __init__(self, group_name, topics_partitions):
        self.group_name = group_name
        self.consumers = {}
        self.assignments = defaultdict(set)
        topics_partitions = sorted(
            [(topic, partition) for topic, partitions in topics_partitions.items() for partition in partitions],
            key=lambda x: (x[0], x[1])
        )

        consumer_index = 0
        consumer_count = 0
        for topic, partition in topics_partitions:
            consumer = f"{group_name}-consumer-{consumer_index}"
            self.consumers[consumer] = consumer_index
            self.assignments[consumer].add((topic, partition))
            consumer_count = len(self.consumers)
            consumer_index = (consumer_index + 1) % (consumer_count if consumer_count > 0 else 1)

    def get_assignment(self, consumer):
        return sorted(self.assignments.get(consumer, set()))

    def get_all_assignments(self):
        result = {}
        for consumer in sorted(self.consumers):
            result[consumer] = self.get_assignment(consumer)
        return result


if __name__ == "__main__":
    topics = {
        "orders": [0, 1, 2],
        "payments": [0, 1],
        "notifications": [0]
    }
    mock_group = ConsumerGroupAssignment("order-service-1", topics)
    for consumer, assignment in mock_group.get_all_assignments().items():
        print(f"{consumer}: {assignment}")

Output

stdout
order-service-1-consumer-0: [('notifications', 0), ('orders', 1)]
order-service-1-consumer-1: [('orders', 0), ('payments', 0)]
order-service-1-consumer-2: [('orders', 2), ('payments', 1)]

How it works

This class model mimics Kafka's round-robin partition assignment strategy without external dependencies. It flattens the topic-partition structure and cycles through consumers by index, using a defaultdict to store partition sets per consumer. The sorted operation on topic-partition pairs ensures deterministic ordering. This is ideal for testing consumer logic offline or validating partitioning behavior before deployment.

Common mistakes

  • Forgetting to sort topics and partitions, causing non-deterministic assignments.
  • Reusing consumer names across groups, leading to overlapping assignments.
  • Not resetting the consumer index when adding new consumers dynamically.

Variations

  1. Use `itertools.cycle` with a list of consumers for cleaner round-robin logic.
  2. Assign partitions in batches (range assignment) instead of one-by-one for larger clusters.

Real-world use cases

  • Unit-testing consumer logic with mock partition assignments before deploying to a live Kafka cluster.
  • Simulating load distribution across consumers in a local development environment without a broker.
  • Validating rebalancing behavior when consumers join or leave a group during integration tests.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Streaming & messaging

Related tutorials and quizzes for this topic.