Streaming & messaging
Kafka-style pub/sub, event consumers, async pipelines, and message-driven workflows.
How to Mock Offset Commit Auto vs Manual in Python
Demonstrates a Kafka-style offset commit function with auto/manual modes and tests it using unittest.mock.patch.
from unittest.mock import Mock, patch
def commit_offsets(topic_partition_offsets, auto_commit=False):
"""Manually commit offsets or simulate auto-commit."""
if auto_commit:
print(f"Auto-committing offsets: {topic_partition_offsets}")
return {"status": "auto_committed"}
print(f"Manuall…How to Mock RabbitMQ Ack Nack Requeue in Python
A mock RabbitMQ channel and consumer that simulates ack, nack, and requeue handling for testing message processing logic without a broker.
import json
from collections import deque
class MockChannel:
def __init__(self):
self.acked = []
self.nacked = []
self.requeued = []
def basic_ack(self, delivery_tag):
self.acked.append(delivery_tag)
def basic_nack(self, delivery_tag, requeue=False):
self.nacked.…How to Mock a Kafka Producer Batch Send in Python
Simulate a Kafka producer in Python that sends batched JSON events with mock partitions and latency for testing streaming pipelines without a real broker.
import json
import random
import time
from datetime import datetime
class MockKafkaProducer:
def __init__(self, topic):
self.topic = topic
self.sent_messages = []
def send(self, value, key=None):
message = {
"topic": self.topic,
"key": key,
"value"…How to Mock a Kafka Rebalance Listener in Python
Simulate Kafka consumer rebalance callbacks (on_partitions_revoked and on_partitions_assigned) with a mock consumer to test listener logic.
import time
from collections import defaultdict
class MockKafkaConsumer:
def __init__(self):
self.assignments = defaultdict(list)
self.rebalances = 0
def assign(self, partitions):
self.rebalances += 1
self.assignments.clear()
for partition in partitions:
s…How to Partition and Order Kafka-Style Messages by Key in Python
Group messages with the same key into ordered buckets using hashing and a defaultdict, mimicking Kafka partition ordering.
from dataclasses import dataclass
from collections import defaultdict
@dataclass
class Message:
key: str
content: str
def partition_and_order(messages, num_partitions=3):
partitions = defaultdict(list)
for msg in messages:
partition_id = hash(msg.key) % num_partitions
partitions[parti…How to Read Redis Streams with XREADGROUP in Python
Read new messages from a Redis stream using a consumer group with XREADGROUP, handling JSON payloads and group creation.
import redis
import json
def read_group_messages(stream_key, group_name, consumer_name, count=10):
r = redis.Redis(host="localhost", port=6379, decode_responses=True)
try:
r.xgroup_create(stream_key, group_name, id="0", mkstream=True)
except redis.exceptions.ResponseError:
pass
messag…How to Serialize and Deserialize JSON Event Payloads in Python
Define an EventPayload class with custom to_json and from_json methods to convert event objects to JSON strings and back, using datetime parsing.
import json
from datetime import datetime
class EventPayload:
def __init__(self, event_id, event_type, timestamp, data):
self.event_id = event_id
self.event_type = event_type
self.timestamp = timestamp
self.data = data
def to_json(self):
return json.dumps({
…How to Simulate RabbitMQ Exchange Routing in Python
Simulate RabbitMQ exchange routing using a nested dict, matching routing keys against patterns like error.* and info.# to return bound queues.
from collections import defaultdict
def route_message(exchanges, exchange_name, routing_key):
"""
Simulate RabbitMQ exchange routing using a nested dict structure.
Returns list of queue names that match the routing key.
"""
queues = exchanges.get(exchange_name, {})
matched = []
for pa…How to Simulate a Micro-Batch Interval Trigger in Python
A dataclass-based mock that emits batch numbers at fixed intervals, mimicking a micro-batch streaming scheduler for testing and development.
import time
from dataclasses import dataclass, field
from typing import List, Callable
@dataclass
class MicroBatchTriggerMock:
batch_interval_seconds: float = 0.5
max_batches: int = 5
_batches_emitted: int = 0
_next_emit_time: float = field(init=False, default=0)
def start(self, on_batch: Callab…How to Stream Join Windowed Mock Topics in Python
Simulates two message topics and joins their events when timestamps fall within a sliding time window using Python generators and deques.
import itertools
import random
import time
from collections import deque
from dataclasses import dataclass, field
@dataclass
class Event:
key: str
value: int
timestamp: float = field(default_factory=time.time)
def generate_topic(prefix, keys, start_time):
while True:
yield Event(
…How to Track Session Windows with Gap Timeout in Python
A Python class that groups events into sessions, closing a session when the gap between events exceeds a timeout threshold.
import time
class SessionWindow:
"""Track sessions with a gap timeout (mock)."""
def __init__(self, timeout_seconds=5):
self.timeout = timeout_seconds
self.session_start = None
self.last_event_time = None
self.event_count = 0
self.events = []
def add_event…How to Wrap Message Attributes in a CloudEvent with Python
Create a minimal CloudEvent dataclass that wraps arbitrary message attributes into a JSON envelope, matching CloudEvents 1.0 spec.
import json
from dataclasses import dataclass, field, asdict
from typing import Any, Dict
from datetime import datetime, timezone
@dataclass
class CloudEvent:
message_attributes: Dict[str, Any] = field(default_factory=dict)
def wrap(self, event_id: str, source: str, event_type: str, data: Any):
self…How to deduplicate messages by ID in Python
Track seen message IDs in a set to skip duplicate messages and store unique content in a dict, with exact output showing which messages were added or skipped.
import time
class MessageStore:
def __init__(self):
self.seen_ids = set()
self.messages = {}
def add(self, message_id, content, timestamp=None):
timestamp = timestamp or time.time()
if message_id in self.seen_ids:
return False
self.seen_ids.add(message_…How to mock RabbitMQ queue binding with routing keys in Python
A mock demonstration of binding a queue to an exchange with multiple routing keys in RabbitMQ using Python and pika, without a real broker connection.
import pika
import sys
def bind_queue_with_routing(channel, queue_name, exchange_name, routing_keys):
"""
Mock RabbitMQ queue binding with routing keys.
Prints the binding configuration instead of connecting to a real broker.
"""
for routing_key in routing_keys:
binding = {
"q…How to mock a CQRS projector read model update in Python
Build a CQRS projector class that maintains denormalized read models by applying domain events in a mock order-processing service.
from dataclasses import dataclass, field
from typing import Dict, List, Optional
@dataclass
class OrderReadModel:
order_id: str
customer_name: str
total: float
status: str = "pending"
items: List[Dict] = field(default_factory=list)
def apply_event(self, event_type: str, payload: Dict) -> Non…Implement a FIFO Message Queue in Python with deque
This code implements a FIFO (first-in-first-out) message queue class using Python's collections.deque, providing enqueue, dequeue, peek, and size operations.
from collections import deque
class MessageQueue:
def __init__(self):
self.queue = deque()
def enqueue(self, message):
self.queue.append(message)
print(f"Enqueued: {message}")
def dequeue(self):
if self.is_empty():
print("Queue is empty, cannot dequeue.")
…Implement a retry queue with visibility timeout in Python
This code simulates a message queue with a visibility timeout, allowing messages to be retried if not deleted before the timeout expires.
import time
from collections import deque
class SimpleQueue:
def __init__(self, visibility_timeout=2):
self.queue = deque()
self.in_flight = {}
self.visibility_timeout = visibility_timeout
def send(self, message):
self.queue.append(message)
def receive(self):
if …Implement the Transactional Outbox Pattern with SQLite in Python
A Python implementation of the transactional outbox pattern using SQLite, ensuring atomic writes of order data and outbox events in a single transaction while supporting reliable message publishing and consumption.
import sqlite3
from dataclasses import dataclass
from datetime import datetime, timezone
import json
@dataclass
class Order:
order_id: str
amount: float
status: str
class TransactionalOutbox:
def __init__(self, db_path=":memory:"):
self.conn = sqlite3.connect(db_path)
self._create_tab…In-Memory PubSub Topic Subscribe Mock in Python
Build a thread-safe in-memory publish/subscribe mock where handlers subscribe to named topics and receive every message published to them.
class PubSub:
def __init__(self):
self.topics = {}
def subscribe(self, topic, callback):
if topic not in self.topics:
self.topics[topic] = []
self.topics[topic].append(callback)
def publish(self, topic, message):
for callback in self.topics.get(topic, []):
…Kafka Consumer Poll Loop Mock in Python
Simulate a Kafka consumer poll loop with a mock class, process messages in batches, and commit offsets to understand streaming consumption patterns.
import time
class MockKafkaConsumer:
def __init__(self, topic, messages):
self.topic = topic
self.messages = list(messages)
self.position = 0
def poll(self, timeout_ms=100):
if self.position >= len(self.messages):
time.sleep(timeout_ms / 1000)
return []…Mock Kafka Consumer Group Partition Assignment in Python
Simulates a Kafka consumer group's round-robin partition assignment with a Python class and prints assignments per consumer.
from collections import defaultdict
class ConsumerGroupAssignment:
def __init__(self, group_name, topics_partitions):
self.group_name = group_name
self.consumers = {}
self.assignments = defaultdict(set)
topics_partitions = sorted(
[(topic, partition) for topic, partiti…Mock NATS queue group load balancing in Python
Simulates a NATS queue group where each message is delivered to exactly one subscriber using random selection with a lightweight mock.
import random
import time
from collections import defaultdict
class MockQueueGroup:
"""Mock a NATS queue group: each message is delivered to exactly one subscriber."""
def __init__(self, subscribers):
self.subscribers = subscribers
def publish(self, message):
receiver = random.choice(se…Mock Protobuf Binary Encoding in Python
Demonstrates a minimal protobuf-like binary encoding and decoding of an event dataclass using varints and length-delimited fields in pure Python.
import struct
from dataclasses import dataclass
@dataclass
class Event:
id: int
user_id: int
action: str
def encode(self) -> bytes:
# Mock protobuf-like binary encoding using varint and length-delimited fields
buf = bytearray()
# field 1: varint id (tag = (1 << 3) | 0 = 8)
…Mock Redis Streams XADD and XREAD in Python
A pure-Python mock of Redis streams that implements basic XADD, XREAD, and XLEN behavior for local testing without a real Redis server.
import redis
import time
import threading
class MockRedisStreams:
def __init__(self):
self.streams = {}
def xadd(self, stream_name, fields):
if stream_name not in self.streams:
self.streams[stream_name] = []
entry_id = f"{time.time_ns()}-{len(self.streams[stream_name])}"
…Browse by section
Each section groups closely related Python snippets.
Streaming & messaging — Python code examples
What you will find here
This page collects streaming & messaging snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.