Reference library

Data pipelines & processing

ETL-style flows, batch transforms, validation, and moving data between formats.

3 matches

Enrich a stream with reference data by key lookup in Python

Uses streamz to join each incoming record to a reference dictionary by name, adding department and level fields or defaults.

streamz streaming join

Python

from streamz import Stream

reference = {"alice": {"dept": "eng", "level": 3}, "bob": {"dept": "sales", "level": 5}}

def enrich(record):
    name = record.get("name")
    ref = reference.get(name)
    joined = dict(record)
    if ref:
        joined.update(ref)
    else:
        joined["dept"] = "unknown"
        joi…

13 0 Open

Data pipelines & processing medium

How to Stream a Large JSONL File Line by Line in Python

Process a large JSON-lines file incrementally using streaming techniques to avoid loading the entire file into memory.

streaming jsonl large-files

Python

import json

def process_large_file(filepath, chunk_size=8192):
    """
    Stream a large JSON-lines file line by line, processing each record
    without loading the entire file into memory.
    """
    total_count = 0
    total_sum = 0
    
    with open(filepath, 'r') as f:
        while True:
            chunk = …

13 0 Open

Data pipelines & processing medium

Implement an Out-of-Order Sort Buffer with a Heap in Python

Buffers out-of-order indices from a stream and emits them in sorted order using a min-heap with a sliding window.

heapq sorting streaming

Python

import heapq
from collections import deque


class OutOfOrderSorter:
    def __init__(self, buffer_size):
        self.buffer_size = buffer_size
        self.buffer = deque(maxlen=buffer_size)
        self.heap = []
        self.next_expected_index = 0
        self.result = []

    def push(self, item):
        heapq.…

12 0 Open

Browse by section

Each section groups closely related Python snippets.