Mock Kafka Consumer Group Partition Assignment in Python
Simulates a Kafka consumer group's round-robin partition assignment with a Python class and prints assignments per consumer.
Python code
41 linesfrom collections import defaultdict
class ConsumerGroupAssignment:
def __init__(self, group_name, topics_partitions):
self.group_name = group_name
self.consumers = {}
self.assignments = defaultdict(set)
topics_partitions = sorted(
[(topic, partition) for topic, partitions in topics_partitions.items() for partition in partitions],
key=lambda x: (x[0], x[1])
)
consumer_index = 0
consumer_count = 0
for topic, partition in topics_partitions:
consumer = f"{group_name}-consumer-{consumer_index}"
self.consumers[consumer] = consumer_index
self.assignments[consumer].add((topic, partition))
consumer_count = len(self.consumers)
consumer_index = (consumer_index + 1) % (consumer_count if consumer_count > 0 else 1)
def get_assignment(self, consumer):
return sorted(self.assignments.get(consumer, set()))
def get_all_assignments(self):
result = {}
for consumer in sorted(self.consumers):
result[consumer] = self.get_assignment(consumer)
return result
if __name__ == "__main__":
topics = {
"orders": [0, 1, 2],
"payments": [0, 1],
"notifications": [0]
}
mock_group = ConsumerGroupAssignment("order-service-1", topics)
for consumer, assignment in mock_group.get_all_assignments().items():
print(f"{consumer}: {assignment}")
Output
order-service-1-consumer-0: [('notifications', 0), ('orders', 1)]
order-service-1-consumer-1: [('orders', 0), ('payments', 0)]
order-service-1-consumer-2: [('orders', 2), ('payments', 1)]
How it works
This class model mimics Kafka's round-robin partition assignment strategy without external dependencies. It flattens the topic-partition structure and cycles through consumers by index, using a defaultdict to store partition sets per consumer. The sorted operation on topic-partition pairs ensures deterministic ordering. This is ideal for testing consumer logic offline or validating partitioning behavior before deployment.
Common mistakes
- Forgetting to sort topics and partitions, causing non-deterministic assignments.
- Reusing consumer names across groups, leading to overlapping assignments.
- Not resetting the consumer index when adding new consumers dynamically.
Variations
- Use `itertools.cycle` with a list of consumers for cleaner round-robin logic.
- Assign partitions in batches (range assignment) instead of one-by-one for larger clusters.
Real-world use cases
- Unit-testing consumer logic with mock partition assignments before deploying to a live Kafka cluster.
- Simulating load distribution across consumers in a local development environment without a broker.
- Validating rebalancing behavior when consumers join or leave a group during integration tests.
Sponsored
More from Streaming & messaging
- At Most Once Fire-and-Forget Mock in Python easy
- Batch Consume Process Commit Pattern in Python medium
- Build a Streaming Messaging Helper in Python easy
- Dead Letter Queue Failed Messages List Mock in Python easy
- Dedupe processed message IDs in Python easy
- Event Envelope with Schema Version Field in Python easy
Keep learning
Related tutorials and quizzes for this topic.