How to Mock a Kafka Rebalance Listener in Python
Simulate Kafka consumer rebalance callbacks (on_partitions_revoked and on_partitions_assigned) with a mock consumer to test listener logic.
Python code
51 linesimport time
from collections import defaultdict
class MockKafkaConsumer:
def __init__(self):
self.assignments = defaultdict(list)
self.rebalances = 0
def assign(self, partitions):
self.rebalances += 1
self.assignments.clear()
for partition in partitions:
self.assignments[partition.topic].append(partition.partition)
print(f"Rebalance #{self.rebalances}: assigned {dict(self.assignments)}")
def poll(self):
time.sleep(0.1)
class RebalanceListener:
def on_partitions_revoked(self, consumer):
print("Partitions revoked, pausing processing...")
consumer.assignments.clear()
def on_partitions_assigned(self, consumer, partitions):
print("New partitions assigned, starting processing...")
consumer.assign(partitions)
class Partition:
def __init__(self, topic, partition):
self.topic = topic
self.partition = partition
def main():
consumer = MockKafkaConsumer()
listener = RebalanceListener()
initial_partitions = [Partition("orders", 0), Partition("orders", 1)]
listener.on_partitions_revoked(consumer)
listener.on_partitions_assigned(consumer, initial_partitions)
new_partitions = [Partition("orders", 1), Partition("orders", 2)]
listener.on_partitions_revoked(consumer)
listener.on_partitions_assigned(consumer, new_partitions)
if __name__ == "__main__":
main()
Output
Partitions revoked, pausing processing...
New partitions assigned, starting processing...
Rebalance #1: assigned {'orders': [0, 1]}
Partitions revoked, pausing processing...
New partitions assigned, starting processing...
Rebalance #2: assigned {'orders': [1, 2]}
How it works
This example models the Kafka consumer rebalance protocol using a lightweight mock. The MockKafkaConsumer tracks assignments in a dictionary and increments a counter on every assignment call, mimicking the assign() method of the real Kafka client. RebalanceListener defines the two standard callbacks: on_partitions_revoked clears the current work state, and on_partitions_assigned calls assign() with the new partition set. The Partition class is a minimal stand-in for the real object holding topic and partition numbers. Calling the callbacks in sequence lets you verify that your listener clears state first, then re-assigns partitions—exactly what happens during a Kafka rebalance.
Common mistakes
- Forgetting that `on_partitions_revoked` fires before the new assignment, so clearing state too late loses messages.
- Using a stale list of partitions after a rebalance instead of the newly provided ones.
- Not resetting the assignments dictionary in `assign()` — state from the previous rebalance leaks into the new one.
- Assuming the mock should use real Kafka partition objects instead of a lightweight stand-in.
Variations
- Use `unittest.mock.MagicMock` to stub the real kafka-python consumer and track calls.
- Call the listener callbacks from a `confluent_kafka` consumer's `subscribe()` with a synthetic assignment.
Real-world use cases
- Verifying partition state resets in a consumer app before automated tests deploy to staging.
- Simulating a broker-triggered rebalance in CI to catch mismanaged offset commits.
- Unit-testing a custom rebalance listener that pauses external side-effects (e.g., DB writes) on revocation.
Sponsored
More from Streaming & messaging
- At Most Once Fire-and-Forget Mock in Python easy
- Batch Consume Process Commit Pattern in Python medium
- Build a Streaming Messaging Helper in Python easy
- Dead Letter Queue Failed Messages List Mock in Python easy
- Dedupe processed message IDs in Python easy
- Event Envelope with Schema Version Field in Python easy
Keep learning
Related tutorials and quizzes for this topic.