Reference library

Streaming & messaging

Kafka-style pub/sub, event consumers, async pipelines, and message-driven workflows.

3 matches

How to Mock a Kafka Rebalance Listener in Python

Simulate Kafka consumer rebalance callbacks (on_partitions_revoked and on_partitions_assigned) with a mock consumer to test listener logic.

kafka rebalance mocking

Python

import time
from collections import defaultdict


class MockKafkaConsumer:
    def __init__(self):
        self.assignments = defaultdict(list)
        self.rebalances = 0

    def assign(self, partitions):
        self.rebalances += 1
        self.assignments.clear()
        for partition in partitions:
            s…

15 0 Open

Streaming & messaging medium

How to Stream Join Windowed Mock Topics in Python

Simulates two message topics and joins their events when timestamps fall within a sliding time window using Python generators and deques.

streaming join generator

Python

import itertools
import random
import time
from collections import deque
from dataclasses import dataclass, field

@dataclass
class Event:
    key: str
    value: int
    timestamp: float = field(default_factory=time.time)

def generate_topic(prefix, keys, start_time):
    while True:
        yield Event(
            …

14 0 Open

Streaming & messaging medium

Implement a retry queue with visibility timeout in Python

This code simulates a message queue with a visibility timeout, allowing messages to be retried if not deleted before the timeout expires.

queue retry visibility-timeout

Python

import time
from collections import deque


class SimpleQueue:
    def __init__(self, visibility_timeout=2):
        self.queue = deque()
        self.in_flight = {}
        self.visibility_timeout = visibility_timeout

    def send(self, message):
        self.queue.append(message)

    def receive(self):
        if …

13 0 Open

Browse by section

Each section groups closely related Python snippets.