Reference library

Data pipelines & processing

ETL-style flows, batch transforms, validation, and moving data between formats.

4 matches

Add a UUID Surrogate Key to Each Row in a CSV with Python

Generate a unique UUID string for every row in a CSV file using the standard-library uuid and csv modules.

import uuid
import csv

def add_surrogate_key(filename):
    with open(filename, newline='') as f_in:
        reader = csv.DictReader(f_in)
        rows = list(reader)

    for row in rows:
        row['surrogate_key'] = str(uuid.uuid4())

    with open(filename, 'w', newline='') as f_out:
        writer = csv.DictWri…

15 0 Open

Data pipelines & processing easy

Generate a Deterministic Hash for Deduplication in Python

Create a stable SHA-256 fingerprint from nested data and file contents to deduplicate records in a data pipeline.

hashing deduplication sha256

Python

import hashlib
import json
from pathlib import Path

def natural_key_hash(data, salt=""):
    """
    Generate a deterministic fingerprint from raw data (dict/list/str).
    Uses JSON canonical-ish serialization with sorted keys and SHA-256.
    """
    canonical = json.dumps(data, sort_keys=True, separators=(",", ":"…

14 0 Open

Data pipelines & processing easy

Generate a Mock CDC Changelog in Python

Simulate a CDC changelog with INSERT, UPDATE, and DELETE operations, timestamps, and record snapshots for testing data pipelines.

cdc changelog mock-data

Python

import json
from datetime import datetime, timedelta


def generate_mock_changelog(records, operations=("INSERT", "UPDATE", "DELETE")):
    """Simulate a CDC changelog from a list of record snapshots."""
    base_time = datetime(2025, 1, 1, 8, 0, 0)
    changelog = []
    for idx, record in enumerate(records):
       …

15 0 Open

Data pipelines & processing easy

How to create a dated snapshot path for a dataset in Python

Generate a versioned directory path combining a base directory, dataset name, and today's date, ready for creating snapshots in data pipelines.

date pathlib datasets

Python

import datetime
import os
from pathlib import Path


def snapshot_path(base_dir: str, dataset_name: str) -> Path:
    """Return a dated snapshot path for a dataset under a base directory."""
    today = datetime.date.today().isoformat()
    return Path(base_dir) / dataset_name / today


if __name__ == "__main__":
    …

15 0 Open

Browse by section

Each section groups closely related Python snippets.