Streaming & messaging
Kafka-style pub/sub, event consumers, async pipelines, and message-driven workflows.
Dedupe processed message IDs in Python
Filters an inbox of messages by removing items whose IDs have already been processed, using a set for fast lookups.
from pathlib import Path
import json
def dedupe_processed_ids(inbox_file: Path, processed_file: Path) -> list:
processed = set(json.loads(processed_file.read_text()))
inbox = json.loads(inbox_file.read_text())
deduped = [item for item in inbox if item["id"] not in processed]
return deduped
if __nam…How to Encode and Decode Avro Data in Python (Roundtrip)
Serialize a Python dict to Avro binary bytes and decode it back using the fastavro-compatible avro library.
import io
import json
from avro.schema import parse
from avro.io import DatumWriter, DatumReader, BinaryEncoder, BinaryDecoder
def avro_roundtrip(schema_json, data):
schema = parse(json.dumps(schema_json))
bytes_writer = io.BytesIO()
encoder = BinaryEncoder(bytes_writer)
writer = DatumWriter(schema)
…How to Implement At-Least-Once Delivery with Acknowledgment in Python
This code demonstrates a mock message broker with at-least-once delivery, including retry logic and acknowledgment after successful processing.
import time
import uuid
from collections import deque
class MockMessageBroker:
def __init__(self):
self.queue = deque()
self.acked = set()
def publish(self, payload: str) -> str:
msg_id = str(uuid.uuid4())
self.queue.append((msg_id, payload))
return msg_id
def po…How to Implement Publish-Subscribe Fanout with Multiple Subscribers in Python
Create a simple publish-subscribe system in Python that broadcasts messages to multiple subscriber callbacks for a given topic.
import time
class PubSub:
def __init__(self):
self.subscribers = {}
def subscribe(self, topic, callback):
if topic not in self.subscribers:
self.subscribers[topic] = []
self.subscribers[topic].append(callback)
def publish(self, topic, message):
if topic in sel…How to Implement an In-Memory Pub/Sub System in Python
This code implements a simple in-memory publish/subscribe system in Python, allowing topics, callbacks, and message broadcasting.
class PubSub:
def __init__(self):
self.topics = {}
def subscribe(self, topic, callback):
if topic not in self.topics:
self.topics[topic] = []
self.topics[topic].append(callback)
return lambda: self.unsubscribe(topic, callback)
def unsubscribe(self, topic, callb…Browse by section
Each section groups closely related Python snippets.
Streaming & messaging — Python code examples
What you will find here
This page collects streaming & messaging snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.