Reference library

Data pipelines & processing

ETL-style flows, batch transforms, validation, and moving data between formats.

6 matches

Add a UUID Surrogate Key to Each Row in a CSV with Python

Generate a unique UUID string for every row in a CSV file using the standard-library uuid and csv modules.

import uuid
import csv

def add_surrogate_key(filename):
    with open(filename, newline='') as f_in:
        reader = csv.DictReader(f_in)
        rows = list(reader)

    for row in rows:
        row['surrogate_key'] = str(uuid.uuid4())

    with open(filename, 'w', newline='') as f_out:
        writer = csv.DictWri…

17 0 Open

Data pipelines & processing easy

How to Convert Data Types in a Python Data Pipeline

Demonstrates a simple Python data pipeline that converts string values to proper types (bool, int, float, datetime) and outputs structured JSON.

data-pipeline type-conversion json

Python

import json
from datetime import datetime

def convert_value(value):
    """Convert string values to appropriate Python types."""
    if value.lower() == "true":
        return True
    if value.lower() == "false":
        return False
    if value.isdigit():
        return int(value)
    try:
        return float(val…

12 0 Open

Data pipelines & processing easy

How to Hash Email Addresses in a PII Masking Pipeline in Python

Replaces every email address in a text string with its SHA-256 hash to protect personally identifiable information (PII).

pii hashing sha256

Python

import hashlib
import re

def hash_email(email: str) -> str:
    """Mask an email address by hashing it with SHA-256."""
    normalized = email.strip().lower()
    return hashlib.sha256(normalized.encode("utf-8")).hexdigest()

def mask_pii_emails(text: str) -> str:
    """Replace all email addresses in text with their…

14 0 Open

Data pipelines & processing easy

How to Safely Coerce Strings to Numbers in Python

A safe conversion function that turns strings into integers or floats, returning a fallback value when conversion fails.

type-conversion robust-parsing data-cleaning

Python

import math

def to_number(value, fallback=None):
    """Safely coerce a string to int or float, returning fallback on failure."""
    if isinstance(value, (int, float)):
        return value
    try:
        # Try int first for clean whole numbers
        return int(value)
    except (ValueError, TypeError):
        …

12 0 Open

Data pipelines & processing easy

How to Validate Data in a Python Pipeline

A helper module to validate common record types — email, positive integer, and non-empty string list — before processing data in a pipeline.

data-validation pipelines type-checking

Python

from typing import Any, Iterable


def is_valid_email(email: str) -> bool:
    """Basic email check: one '@', no spaces, dot after '@'."""
    if "@" not in email or " " in email:
        return False
    local, _, domain = email.partition("@")
    return bool(local) and "." in domain


def is_positive_int(value: Any)…

12 0 Open

Data pipelines & processing easy

How to shard output by primary key hash mod N in Python

This code computes a consistent shard index for any primary key string using an MD5 hash mod the number of shards, enabling stable key-based data distribution.

hashing sharding hashlib

Python

import hashlib

def shard_id(primary_key: str, num_shards: int) -> int:
    """Return the shard index for a primary key using MD5 hash mod N."""
    digest = hashlib.md5(primary_key.encode("utf-8")).hexdigest()
    hash_int = int(digest, 16)
    return hash_int % num_shards

if __name__ == "__main__":
    keys = ["use…

11 0 Open

Browse by section

Each section groups closely related Python snippets.