Data pipelines & processing
ETL-style flows, batch transforms, validation, and moving data between formats.
Group Python Events into Sessions with a Gap Timeout
Groups timestamped events into sessions, starting a new session when the time gap exceeds a specified timeout.
from itertools import groupby
from datetime import datetime, timedelta
def session_window_group(events, gap_seconds=300):
"""Group events into sessions where gap > gap_seconds starts a new session."""
if not events:
return []
events = sorted(events, key=lambda x: x[0])
sessions = []
c…Trigger a Pipeline When a New File Appears in a Directory
Poll a directory every 0.5 seconds and return the name of the first new file that appears, or None after a timeout.
import time
from pathlib import Path
def watch_for_file(directory: str, interval: float = 0.5, timeout: float = 10.0) -> str | None:
"""Poll a directory and trigger when a new file appears."""
watch_dir = Path(directory)
watch_dir.mkdir(exist_ok=True)
known_files = set(watch_dir.iterdir())
s…Browse by section
Each section groups closely related Python snippets.
Data pipelines & processing — Python code examples
What you will find here
This page collects data pipelines & processing snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.