How to Mock a Kafka Rebalance Listener in Python

Simulate Kafka consumer rebalance callbacks (on_partitions_revoked and on_partitions_assigned) with a mock consumer to test listener logic.

Medium Python 3.9+ Aug 9, 2026 Streaming & messaging 15 views 0 copies

Python code

51 lines
Python 3.9+
import time
from collections import defaultdict


class MockKafkaConsumer:
    def __init__(self):
        self.assignments = defaultdict(list)
        self.rebalances = 0

    def assign(self, partitions):
        self.rebalances += 1
        self.assignments.clear()
        for partition in partitions:
            self.assignments[partition.topic].append(partition.partition)
        print(f"Rebalance #{self.rebalances}: assigned {dict(self.assignments)}")

    def poll(self):
        time.sleep(0.1)


class RebalanceListener:
    def on_partitions_revoked(self, consumer):
        print("Partitions revoked, pausing processing...")
        consumer.assignments.clear()

    def on_partitions_assigned(self, consumer, partitions):
        print("New partitions assigned, starting processing...")
        consumer.assign(partitions)


class Partition:
    def __init__(self, topic, partition):
        self.topic = topic
        self.partition = partition


def main():
    consumer = MockKafkaConsumer()
    listener = RebalanceListener()

    initial_partitions = [Partition("orders", 0), Partition("orders", 1)]
    listener.on_partitions_revoked(consumer)
    listener.on_partitions_assigned(consumer, initial_partitions)

    new_partitions = [Partition("orders", 1), Partition("orders", 2)]
    listener.on_partitions_revoked(consumer)
    listener.on_partitions_assigned(consumer, new_partitions)


if __name__ == "__main__":
    main()

Output

stdout
Partitions revoked, pausing processing...
New partitions assigned, starting processing...
Rebalance #1: assigned {'orders': [0, 1]}
Partitions revoked, pausing processing...
New partitions assigned, starting processing...
Rebalance #2: assigned {'orders': [1, 2]}

How it works

This example models the Kafka consumer rebalance protocol using a lightweight mock. The MockKafkaConsumer tracks assignments in a dictionary and increments a counter on every assignment call, mimicking the assign() method of the real Kafka client. RebalanceListener defines the two standard callbacks: on_partitions_revoked clears the current work state, and on_partitions_assigned calls assign() with the new partition set. The Partition class is a minimal stand-in for the real object holding topic and partition numbers. Calling the callbacks in sequence lets you verify that your listener clears state first, then re-assigns partitions—exactly what happens during a Kafka rebalance.

Common mistakes

  • Forgetting that `on_partitions_revoked` fires before the new assignment, so clearing state too late loses messages.
  • Using a stale list of partitions after a rebalance instead of the newly provided ones.
  • Not resetting the assignments dictionary in `assign()` — state from the previous rebalance leaks into the new one.
  • Assuming the mock should use real Kafka partition objects instead of a lightweight stand-in.

Variations

  1. Use `unittest.mock.MagicMock` to stub the real kafka-python consumer and track calls.
  2. Call the listener callbacks from a `confluent_kafka` consumer's `subscribe()` with a synthetic assignment.

Real-world use cases

  • Verifying partition state resets in a consumer app before automated tests deploy to staging.
  • Simulating a broker-triggered rebalance in CI to catch mismanaged offset commits.
  • Unit-testing a custom rebalance listener that pauses external side-effects (e.g., DB writes) on revocation.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Streaming & messaging

Related tutorials and quizzes for this topic.