Kafka Consumer Poll Loop Mock in Python

Simulate a Kafka consumer poll loop with a mock class, process messages in batches, and commit offsets to understand streaming consumption patterns.

Medium Python 3.9+ Aug 9, 2026 Streaming & messaging 13 views 0 copies

kafka streaming mock poll consumer

Python code

38 lines

Python 3.9+

import time

class MockKafkaConsumer:
    def __init__(self, topic, messages):
        self.topic = topic
        self.messages = list(messages)
        self.position = 0

    def poll(self, timeout_ms=100):
        if self.position >= len(self.messages):
            time.sleep(timeout_ms / 1000)
            return []
        batch = self.messages[self.position:self.position + 3]
        self.position += len(batch)
        return batch

    def commit(self):
        print(f"Committed offset: {self.position}")

consumer = MockKafkaConsumer("orders", [
    {"id": 1, "item": "apple"},
    {"id": 2, "item": "banana"},
    {"id": 3, "item": "cherry"},
    {"id": 4, "item": "date"},
    {"id": 5, "item": "elderberry"},
])

if __name__ == "__main__":
    total_processed = 0
    while True:
        records = consumer.poll(timeout_ms=50)
        if not records:
            break
        for record in records:
            print(f"Processing {record['id']}: {record['item']}")
            total_processed += 1
        consumer.commit()
    print(f"Total processed: {total_processed}")

Output

stdout

Processing 1: apple
Committed offset: 3
Processing 2: banana
Processing 3: cherry
Committed offset: 3
Processing 4: date
Processing 5: elderberry
Committed offset: 5
Total processed: 5

How it works

The MockKafkaConsumer mimics a Kafka consumer by returning batches of up to 3 messages per poll call, tracking its position in the message list. The main loop polls repeatedly until an empty batch signals the end of the stream, processing each record and committing the offset after each batch. The time.sleep simulates the blocking behavior of a real poll when no messages are available. This pattern mirrors production Kafka consumers that continuously poll for new records and commit offsets to track progress.

Common mistakes

Forgetting to break the loop when poll returns an empty batch, causing an infinite loop
Assuming poll returns all messages at once instead of in batches
Committing offsets too frequently, which kills throughput in real Kafka consumers
Not handling consumer rebalancing in real Kafka consumers when partitions move between instances

Variations

Use `concurrent.futures` with a ThreadPoolExecutor to process records in parallel within each batch
Implement an `order_by` or filtering step before processing to only handle records matching certain criteria

Real-world use cases

Building a data pipeline that reads events from Kafka and writes them to a data warehouse in near real-time.
Creating a feature service that consumes user interaction events to update recommendation models.
Developing a notification system that polls Kafka for order events and sends email confirmations.

Kafka Consumer Poll Loop Mock in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Streaming & messaging

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Streaming & messaging

Keep learning

Tutorials

Quizzes