Reference library

Data pipelines & processing

ETL-style flows, batch transforms, validation, and moving data between formats.

3 matches

How to Count Events by Minute with a Tumbling Window in Python

Group timestamps into fixed 60-second tumbling windows and count events per bucket using a dict.

from collections import defaultdict
from datetime import datetime, timedelta


def tumbling_window_count(events, window_seconds=60):
    buckets = defaultdict(int)
    for event in events:
        ts = datetime.fromisoformat(event["timestamp"])
        bucket_start = ts - timedelta(seconds=ts.second % window_seconds,
…

13 0 Open

Data pipelines & processing medium

How to Stream a Large JSONL File Line by Line in Python

Process a large JSON-lines file incrementally using streaming techniques to avoid loading the entire file into memory.

streaming jsonl large-files

Python

import json

def process_large_file(filepath, chunk_size=8192):
    """
    Stream a large JSON-lines file line by line, processing each record
    without loading the entire file into memory.
    """
    total_count = 0
    total_sum = 0
    
    with open(filepath, 'r') as f:
        while True:
            chunk = …

13 0 Open

Data pipelines & processing medium

Python Exponential Backoff Retry Example

Retry a flaky function with exponential backoff and jitter-free delays, printing each attempt and finally returning the successful result.

retry backoff exception-handling

Python

import random
import time


def flaky_function():
    if random.random() < 0.6:
        raise ConnectionError("Temporary network error")
    return "success"


def retry_with_exponential_backoff(func, max_retries=5, base_delay=1.0):
    for attempt in range(max_retries + 1):
        try:
            return func()
    …

16 0 Open

Browse by section

Each section groups closely related Python snippets.