Reference library

Streaming & messaging

Kafka-style pub/sub, event consumers, async pipelines, and message-driven workflows.

5 matches

How to Mock Kafka Topic Partitions with a Python dict of lists

Mocks a Kafka topic and its partitions using a defaultdict of lists to simulate message production, consumption, and per-partition counts.

kafka mock partitions

Python

from collections import defaultdict

class KafkaTopicPartitionMock:
    """A simple mock for Kafka topic-partition assignment using dict of lists."""

    def __init__(self, topic):
        self.topic = topic
        self.partitions = defaultdict(list)  # partition_id -> list of messages

    def produce(self, message…

15 0 Open

Streaming & messaging medium

How to Mock a Kafka Producer Batch Send in Python

Simulate a Kafka producer in Python that sends batched JSON events with mock partitions and latency for testing streaming pipelines without a real broker.

kafka mock streaming

Python

import json
import random
import time
from datetime import datetime


class MockKafkaProducer:
    def __init__(self, topic):
        self.topic = topic
        self.sent_messages = []

    def send(self, value, key=None):
        message = {
            "topic": self.topic,
            "key": key,
            "value"…

13 0 Open

Streaming & messaging medium

How to Mock a Kafka Rebalance Listener in Python

Simulate Kafka consumer rebalance callbacks (on_partitions_revoked and on_partitions_assigned) with a mock consumer to test listener logic.

kafka rebalance mocking

Python

import time
from collections import defaultdict


class MockKafkaConsumer:
    def __init__(self):
        self.assignments = defaultdict(list)
        self.rebalances = 0

    def assign(self, partitions):
        self.rebalances += 1
        self.assignments.clear()
        for partition in partitions:
            s…

15 0 Open

Streaming & messaging easy

How to Partition and Order Kafka-Style Messages by Key in Python

Group messages with the same key into ordered buckets using hashing and a defaultdict, mimicking Kafka partition ordering.

streaming partitioning kafka-pattern

Python

from dataclasses import dataclass
from collections import defaultdict

@dataclass
class Message:
    key: str
    content: str

def partition_and_order(messages, num_partitions=3):
    partitions = defaultdict(list)
    for msg in messages:
        partition_id = hash(msg.key) % num_partitions
        partitions[parti…

14 0 Open

Streaming & messaging medium

Mock Kafka Consumer Group Partition Assignment in Python

Simulates a Kafka consumer group's round-robin partition assignment with a Python class and prints assignments per consumer.

kafka consumer-group partition-assignment

Python

from collections import defaultdict


class ConsumerGroupAssignment:
    def __init__(self, group_name, topics_partitions):
        self.group_name = group_name
        self.consumers = {}
        self.assignments = defaultdict(set)
        topics_partitions = sorted(
            [(topic, partition) for topic, partiti…

14 0 Open

Browse by section

Each section groups closely related Python snippets.