Reference library

Data pipelines & processing

ETL-style flows, batch transforms, validation, and moving data between formats.

3 matches

Add a UUID Surrogate Key to Each Row in a CSV with Python

Generate a unique UUID string for every row in a CSV file using the standard-library uuid and csv modules.

import uuid
import csv

def add_surrogate_key(filename):
    with open(filename, newline='') as f_in:
        reader = csv.DictReader(f_in)
        rows = list(reader)

    for row in rows:
        row['surrogate_key'] = str(uuid.uuid4())

    with open(filename, 'w', newline='') as f_out:
        writer = csv.DictWri…

14 0 Open

Data pipelines & processing easy

How to Clean and Format Data in Python

This code loads JSON data, cleans records by removing empty fields and normalizing text, then summarizes the results with counts and unique keys.

json data cleaning data pipelines

Python

import json
from pathlib import Path


def load_data(filepath: str) -> dict:
    """Load JSON data from a file."""
    with Path(filepath).open("r", encoding="utf-8") as f:
        return json.load(f)


def clean_records(records: list[dict]) -> list[dict]:
    """Remove empty fields and normalize text to lowercase."""…

13 0 Open

Data pipelines & processing medium

How to Validate Fact Table Grain Row Counts in Python

Validate fact table grain by checking dimension key references, unique grain combinations, duplicate rows, and dimension cardinality from a CSV file.

csv data validation etl

Python

import csv
import hashlib
from pathlib import Path


def validate_fact_grain(fact_file: Path, expected_dim_keys: dict[str, set[str]]) -> dict:
    """
    Validate fact table grain by checking each row's dimension keys exist
    in expected dimension tables and row count consistency.
    """
    dim_references = {}
  …

13 0 Open

Browse by section

Each section groups closely related Python snippets.