Reference library

Data pipelines & processing

ETL-style flows, batch transforms, validation, and moving data between formats.

3 matches

Count Records Processed per Category in Python

Use a Counter dictionary to track how many records of each type (ok, error, retry) were processed in a data pipeline.

from collections import Counter
import random

processed_counter = Counter()

def process_records(records):
    for record in records:
        processed_counter[record] += 1
    return len(records)

if __name__ == "__main__":
    sample_records = [random.choice(["ok", "error", "retry"]) for _ in range(10)]
    print(f…

15 0 Open

Data pipelines & processing easy

How to Compress Pipeline Output Gzip Per Partition in Python

Compress each partition of pipeline output into a separate gzip file and verify the compressed data by reading it back.

gzip compression pipeline

Python

import gzip
import io
import random
from pathlib import Path


def compress_partition(partition_data: list[str], output_path: Path) -> int:
    """Compress a partition of data to a gzip file, returns bytes written."""
    with gzip.open(output_path, 'wt', encoding='utf-8') as f:
        f.writelines(partition_data)
  …

13 0 Open

Data pipelines & processing easy

How to Hash Email Addresses in a PII Masking Pipeline in Python

Replaces every email address in a text string with its SHA-256 hash to protect personally identifiable information (PII).

pii hashing sha256

Python

import hashlib
import re

def hash_email(email: str) -> str:
    """Mask an email address by hashing it with SHA-256."""
    normalized = email.strip().lower()
    return hashlib.sha256(normalized.encode("utf-8")).hexdigest()

def mask_pii_emails(text: str) -> str:
    """Replace all email addresses in text with their…

14 0 Open

Browse by section

Each section groups closely related Python snippets.