Streaming & messaging
Kafka-style pub/sub, event consumers, async pipelines, and message-driven workflows.
Implement the Transactional Outbox Pattern with SQLite in Python
A Python implementation of the transactional outbox pattern using SQLite, ensuring atomic writes of order data and outbox events in a single transaction while supporting reliable message publishing and consumption.
import sqlite3
from dataclasses import dataclass
from datetime import datetime, timezone
import json
@dataclass
class Order:
order_id: str
amount: float
status: str
class TransactionalOutbox:
def __init__(self, db_path=":memory:"):
self.conn = sqlite3.connect(db_path)
self._create_tab…Kafka Consumer Poll Loop Mock in Python
Simulate a Kafka consumer poll loop with a mock class, process messages in batches, and commit offsets to understand streaming consumption patterns.
import time
class MockKafkaConsumer:
def __init__(self, topic, messages):
self.topic = topic
self.messages = list(messages)
self.position = 0
def poll(self, timeout_ms=100):
if self.position >= len(self.messages):
time.sleep(timeout_ms / 1000)
return []…Mock Kafka Consumer Group Partition Assignment in Python
Simulates a Kafka consumer group's round-robin partition assignment with a Python class and prints assignments per consumer.
from collections import defaultdict
class ConsumerGroupAssignment:
def __init__(self, group_name, topics_partitions):
self.group_name = group_name
self.consumers = {}
self.assignments = defaultdict(set)
topics_partitions = sorted(
[(topic, partition) for topic, partiti…Mock NATS queue group load balancing in Python
Simulates a NATS queue group where each message is delivered to exactly one subscriber using random selection with a lightweight mock.
import random
import time
from collections import defaultdict
class MockQueueGroup:
"""Mock a NATS queue group: each message is delivered to exactly one subscriber."""
def __init__(self, subscribers):
self.subscribers = subscribers
def publish(self, message):
receiver = random.choice(se…Mock Protobuf Binary Encoding in Python
Demonstrates a minimal protobuf-like binary encoding and decoding of an event dataclass using varints and length-delimited fields in pure Python.
import struct
from dataclasses import dataclass
@dataclass
class Event:
id: int
user_id: int
action: str
def encode(self) -> bytes:
# Mock protobuf-like binary encoding using varint and length-delimited fields
buf = bytearray()
# field 1: varint id (tag = (1 << 3) | 0 = 8)
…Mock Redis Streams XADD and XREAD in Python
A pure-Python mock of Redis streams that implements basic XADD, XREAD, and XLEN behavior for local testing without a real Redis server.
import redis
import time
import threading
class MockRedisStreams:
def __init__(self):
self.streams = {}
def xadd(self, stream_name, fields):
if stream_name not in self.streams:
self.streams[stream_name] = []
entry_id = f"{time.time_ns()}-{len(self.streams[stream_name])}"
…Mock Watermark Late Event Side Output in Python
Simulates watermarking in a streaming pipeline by classifying events as on-time or late using timestamps and delays.
from datetime import datetime, timedelta
from typing import List, Tuple
def watermark_mock(
events: List[Tuple[datetime, str]], watermark_delay: timedelta, max_delay: timedelta
) -> Tuple[List[Tuple[datetime, str]], List[Tuple[datetime, str]]]:
"""Simulate watermarking: events arriving on time vs. late by ch…Redis Pub/Sub Channel Subscribe Mock in Python
A lightweight in-memory mock of Redis pub/sub that lets you subscribe to channels, publish messages, and verify handler behavior in tests without a real Redis server.
class MockRedisPubSub:
def __init__(self):
self.channels = {}
def subscribe(self, channel):
if channel not in self.channels:
self.channels[channel] = []
return self.channels[channel]
def publish(self, channel, message):
if channel in self.channels:
…Sliding Window Average with Deque in Python
Computes the running average of a sliding window over streaming numbers using a collections.deque for O(1) pop-left operations.
from collections import deque
class SlidingAverage:
def __init__(self, window_size):
self.window_size = window_size
self.window = deque()
self.total = 0
def add(self, value):
self.window.append(value)
self.total += value
if len(self.window) > self.window_size:
…Using the retained message flag in MQTT with Python
This script subscribes to an MQTT topic and prints the retained flag for each received message, demonstrating how to distinguish retained messages from normal ones.
import paho.mqtt.client as mqtt
def on_connect(client, userdata, flags, rc):
print(f"Connected with result code {rc}")
# Subscribe to a topic and check retained flag
client.subscribe("test/retained")
print("Subscribed to test/retained")
def on_message(client, userdata, msg):
# msg.retain is the M…Browse by section
Each section groups closely related Python snippets.
Streaming & messaging — Python code examples
What you will find here
This page collects streaming & messaging snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.