Python Code
Samples
Medium snippets you can copy, study, and run in the browser editor.
How to Stream Large CSV Files in Python
Process a large CSV file in memory-efficient chunks using Python's csv module, yielding batches of rows instead of loading everything at once.
import csv
from pathlib import Path
def process_csv_in_chunks(file_path, chunk_size=1000):
"""Yield rows from a large CSV file in chunks without loading all into memory."""
with open(file_path, 'r', newline='') as f:
reader = csv.DictReader(f)
chunk = []
for row in reader:
…Read Parquet-Like Columnar CSV Chunks in Python
A Python generator that reads a CSV file column-by-column, yielding dictionary chunks where each key points to a list of values—mirroring how Parquet stores data columnar.
```python
import csv
from pathlib import Path
from typing import Iterator, List
def read_parquet_like_columnar(csv_path: str, column_names: List[str], chunk_size: int = 2) -> Iterator[dict]:
"""Read CSV data in columnar chunks, similar to how parquet stores columns."""
csv_file = Path(csv_path)
with csv_f…How to stream parse JSON arrays in Python
This code demonstrates two generators: one that streams a JSON array as individual chunks, and another that incrementally parses those chunks into Python objects using json.JSONDecoder.
import json
def json_array_stream(items):
"""Generator that yields JSON-encoded values one at a time."""
yield "["
for i, item in enumerate(items):
if i > 0:
yield ","
yield json.dumps(item)
yield "]"
def parse_json_stream(stream):
"""Consumes a stream of JSON fragme…Enrich a stream with reference data by key lookup in Python
Uses streamz to join each incoming record to a reference dictionary by name, adding department and level fields or defaults.
from streamz import Stream
reference = {"alice": {"dept": "eng", "level": 3}, "bob": {"dept": "sales", "level": 5}}
def enrich(record):
name = record.get("name")
ref = reference.get(name)
joined = dict(record)
if ref:
joined.update(ref)
else:
joined["dept"] = "unknown"
joi…How to Stream a Large JSONL File Line by Line in Python
Process a large JSON-lines file incrementally using streaming techniques to avoid loading the entire file into memory.
import json
def process_large_file(filepath, chunk_size=8192):
"""
Stream a large JSON-lines file line by line, processing each record
without loading the entire file into memory.
"""
total_count = 0
total_sum = 0
with open(filepath, 'r') as f:
while True:
chunk = …Implement an Out-of-Order Sort Buffer with a Heap in Python
Buffers out-of-order indices from a stream and emits them in sorted order using a min-heap with a sliding window.
import heapq
from collections import deque
class OutOfOrderSorter:
def __init__(self, buffer_size):
self.buffer_size = buffer_size
self.buffer = deque(maxlen=buffer_size)
self.heap = []
self.next_expected_index = 0
self.result = []
def push(self, item):
heapq.…How to Mock a Chunked Encoding Streaming Response in Python
Build a local mock HTTP server with Python's http.server that streams a chunked-encoded response with a 0.5s delay per chunk.
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import time
class ChunkedHandler(BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
def do_GET(self):
self.send_response(200)
self.send_header("Content-Type", "text/plain")
self.send_header("Transfer-Encoding", "c…How to mock Server-Sent Events (SSE) in Python
A minimal HTTP server that streams Server-Sent Events to clients, perfect for testing and development.
from http.server import HTTPServer, BaseHTTPRequestHandler
import threading
import time
MESSAGES = iter([
"data: Hello world\n\n",
"data: Second message\n\n",
"event: custom\n",
"data: Custom event payload\n\n",
"data: Final message\n\n"
])
class SSEHandler(BaseHTTPRequestHandler):
def do_GET…Batch Consume Process Commit Pattern in Python
A mock batch processor that accumulates items in a queue, processes full batches, commits successful or failed results, and flushes remaining items.
import random
import threading
import time
from collections import deque
class MockBatchProcessor:
def __init__(self, process_func, commit_func, batch_size=5):
self.queue = deque()
self.batch_size = batch_size
self.process_func = process_func
self.commit_func = commit_func
de…How to Aggregate Periodic Snapshot Data in Python
Generates mock snapshot data and groups values into periods to compute average aggregates with Python's standard library.
import random
from collections import defaultdict
def snapshot_aggregate(n=10, period=3):
data = defaultdict(list)
for i in range(n):
key = f"item_{i % period}"
data[key].append(random.randint(1, 100))
return dict(data)
def aggregate_periodic(snapshots, period=3):
result = {}
for …How to Build a Flow Control Credit Window in Python
A Python class that reserves, confirms, releases, and settles credit to limit message flow and prevent overload in streaming pipelines.
class CreditWindow:
def __init__(self, max_credit=1000):
self.max_credit = max_credit
self.used_credit = 0
self.pending_credit = 0
def try_reserve(self, amount):
available = self.max_credit - self.used_credit - self.pending_credit
if available >= amount:
…How to Encode and Decode Avro Data in Python (Roundtrip)
Serialize a Python dict to Avro binary bytes and decode it back using the fastavro-compatible avro library.
import io
import json
from avro.schema import parse
from avro.io import DatumWriter, DatumReader, BinaryEncoder, BinaryDecoder
def avro_roundtrip(schema_json, data):
schema = parse(json.dumps(schema_json))
bytes_writer = io.BytesIO()
encoder = BinaryEncoder(bytes_writer)
writer = DatumWriter(schema)
…How to Mock Offset Commit Auto vs Manual in Python
Demonstrates a Kafka-style offset commit function with auto/manual modes and tests it using unittest.mock.patch.
from unittest.mock import Mock, patch
def commit_offsets(topic_partition_offsets, auto_commit=False):
"""Manually commit offsets or simulate auto-commit."""
if auto_commit:
print(f"Auto-committing offsets: {topic_partition_offsets}")
return {"status": "auto_committed"}
print(f"Manuall…How to Mock a Kafka Producer Batch Send in Python
Simulate a Kafka producer in Python that sends batched JSON events with mock partitions and latency for testing streaming pipelines without a real broker.
import json
import random
import time
from datetime import datetime
class MockKafkaProducer:
def __init__(self, topic):
self.topic = topic
self.sent_messages = []
def send(self, value, key=None):
message = {
"topic": self.topic,
"key": key,
"value"…How to Mock a Kafka Rebalance Listener in Python
Simulate Kafka consumer rebalance callbacks (on_partitions_revoked and on_partitions_assigned) with a mock consumer to test listener logic.
import time
from collections import defaultdict
class MockKafkaConsumer:
def __init__(self):
self.assignments = defaultdict(list)
self.rebalances = 0
def assign(self, partitions):
self.rebalances += 1
self.assignments.clear()
for partition in partitions:
s…How to Stream Join Windowed Mock Topics in Python
Simulates two message topics and joins their events when timestamps fall within a sliding time window using Python generators and deques.
import itertools
import random
import time
from collections import deque
from dataclasses import dataclass, field
@dataclass
class Event:
key: str
value: int
timestamp: float = field(default_factory=time.time)
def generate_topic(prefix, keys, start_time):
while True:
yield Event(
…How to Track Session Windows with Gap Timeout in Python
A Python class that groups events into sessions, closing a session when the gap between events exceeds a timeout threshold.
import time
class SessionWindow:
"""Track sessions with a gap timeout (mock)."""
def __init__(self, timeout_seconds=5):
self.timeout = timeout_seconds
self.session_start = None
self.last_event_time = None
self.event_count = 0
self.events = []
def add_event…Kafka Consumer Poll Loop Mock in Python
Simulate a Kafka consumer poll loop with a mock class, process messages in batches, and commit offsets to understand streaming consumption patterns.
import time
class MockKafkaConsumer:
def __init__(self, topic, messages):
self.topic = topic
self.messages = list(messages)
self.position = 0
def poll(self, timeout_ms=100):
if self.position >= len(self.messages):
time.sleep(timeout_ms / 1000)
return []…Mock Watermark Late Event Side Output in Python
Simulates watermarking in a streaming pipeline by classifying events as on-time or late using timestamps and delays.
from datetime import datetime, timedelta
from typing import List, Tuple
def watermark_mock(
events: List[Tuple[datetime, str]], watermark_delay: timedelta, max_delay: timedelta
) -> Tuple[List[Tuple[datetime, str]], List[Tuple[datetime, str]]]:
"""Simulate watermarking: events arriving on time vs. late by ch…Approximate Distinct Count in Python with HyperLogLog
Mock a large data stream and estimate the number of distinct items with a HyperLogLog-style probabilistic counter to save memory.
import random
import string
from collections import Counter
import math
class ApproxCountDistinct:
def __init__(self, num_buckets=16):
self.num_buckets = num_buckets
self.max_zeros = [0] * num_buckets
def _hash(self, item):
# Simple string hash to a 32-bit integer
h = …How to Implement a Streaming Watermark in Python
Mock structured streaming watermarks in Python to track late event times and compute a watermark for windowed processing.
from datetime import datetime, timedelta
import time
class StreamingWatermark:
"""Mock watermark tracker for structured streaming."""
def __init__(self, watermark_delay_seconds):
self.watermark_delay = timedelta(seconds=watermark_delay_seconds)
self.max_event_time = None
def observe_even…How to Mock Spark Streaming Micro-Batches in Python
Simulate Spark's micro-batch streaming with a simple deque-based class that collects events over time and processes them in timed batches.
import time
from collections import deque
from datetime import datetime
class MicroBatchStream:
def __init__(self, batch_interval_sec=2):
self.batch_interval = batch_interval_sec
self.source = deque()
self.processed = []
def add_events(self, events):
self.source.extend(events…How to Mock and Test a Rate-Limited Source Stream in Python
Build a class that rate-limits emitted items using a sliding window and test it with a simulated stream in Python.
import time
from collections import deque
class RateLimitedSource:
def __init__(self, max_rate, window=1.0):
self.max_rate = max_rate
self.window = window
self._timestamps = deque()
def emit(self, item):
now = time.monotonic()
while self._timestamps and self._timestam…How to implement a tumbling window aggregation in Python
Build a mock tumbling window aggregator in Python that groups streaming events into fixed time intervals and computes count, sum, and average per window.
import time
from collections import deque
class TumblingWindow:
def __init__(self, duration_seconds):
self.duration = duration_seconds
self.buffer = deque()
self.window_start = None
def add(self, item):
current_time = time.time()
if self.window_start is None:
…Browse by section
Each section groups closely related Python snippets.
Guide: free Python code samples library
Copy-ready Python snippets for learners and developers
PythonSkillset code samples are short, focused examples organised by topic and difficulty. Every snippet is server-rendered HTML — readable by search engines and easy to copy. Open any sample, read the notes, copy the code, then press Try in editor to run it in the browser with Pyodide.
How to use this library
- Pick a topic section — strings, lists, files, functions, and more
- Open a sample, read How it works, and copy the code block
- Run it in the IDE, tweak values, then take a related quiz or tutorial lesson
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.