Reference library

Data pipelines & processing

ETL-style flows, batch transforms, validation, and moving data between formats.

5 matches

Attach Source File Metadata to Records in Python

Add a source filename field to each record in a list by merging a new key into every dictionary using a dict unpacking comprehension.

lineage metadata dict-unpacking

Python

from pathlib import Path
import json

def attach_source_metadata(records, source_file):
    """Attach source filename metadata to each record."""
    return [
        {**record, "source": Path(source_file).name}
        for record in records
    ]

if __name__ == "__main__":
    source = "/data/raw/customers.csv"
    …

15 0 Open

Data pipelines & processing easy

How to Build a Simple Data Pipeline in Python

A beginner-friendly data pipeline that loads JSON, filters records by a field value, and aggregates counts per category.

pipeline json aggregation

Python

import json
from pathlib import Path


def load_json(filepath: str | Path) -> list[dict]:
    """Load a JSON file containing a list of records."""
    with Path(filepath).open("r", encoding="utf-8") as f:
        return json.load(f)


def filter_records(records: list[dict], field: str, value) -> list[dict]:
    """Kee…

10 0 Open

Data pipelines & processing easy

How to Partition Output Files by Date Key in Python

Group output files into a dictionary partitioned by a YYYYMMDD date key extracted from the filename prefix.

file-partitioning date-key pathlib

Python

from pathlib import Path
from collections import defaultdict

def partition_files_by_date(directory: str) -> dict:
    """Partition output files by date key extracted from filename (YYYYMMDD prefix)."""
    path = Path(directory)
    partitions = defaultdict(list)
    
    for file in path.iterdir():
        if file.i…

13 0 Open

Data pipelines & processing easy

How to Process CSV Data in Python with a Data Helper

Build a beginner-friendly data helper in Python that loads a CSV file, filters rows by a condition, and summarizes numeric fields.

csv data-processing pathlib

Python

import csv
from pathlib import Path

DATA = [
    {"name": "Alice", "score": 88, "passed": True},
    {"name": "Bob", "score": 42, "passed": False},
    {"name": "Carol", "score": 95, "passed": True},
]


def load_csv(file_path: Path) -> list[dict]:
    with file_path.open(newline="", encoding="utf-8") as f:
        r…

12 0 Open

Data pipelines & processing easy

How to create a dated snapshot path for a dataset in Python

Generate a versioned directory path combining a base directory, dataset name, and today's date, ready for creating snapshots in data pipelines.

date pathlib datasets

Python

import datetime
import os
from pathlib import Path


def snapshot_path(base_dir: str, dataset_name: str) -> Path:
    """Return a dated snapshot path for a dataset under a base directory."""
    today = datetime.date.today().isoformat()
    return Path(base_dir) / dataset_name / today


if __name__ == "__main__":
    …

14 0 Open

Browse by section

Each section groups closely related Python snippets.