Data pipelines & processing
ETL-style flows, batch transforms, validation, and moving data between formats.
Add a UUID Surrogate Key to Each Row in a CSV with Python
Generate a unique UUID string for every row in a CSV file using the standard-library uuid and csv modules.
import uuid
import csv
def add_surrogate_key(filename):
with open(filename, newline='') as f_in:
reader = csv.DictReader(f_in)
rows = list(reader)
for row in rows:
row['surrogate_key'] = str(uuid.uuid4())
with open(filename, 'w', newline='') as f_out:
writer = csv.DictWri…Attach Source File Metadata to Records in Python
Add a source filename field to each record in a list by merging a new key into every dictionary using a dict unpacking comprehension.
from pathlib import Path
import json
def attach_source_metadata(records, source_file):
"""Attach source filename metadata to each record."""
return [
{**record, "source": Path(source_file).name}
for record in records
]
if __name__ == "__main__":
source = "/data/raw/customers.csv"
…Build a Python Utility That Detects Duplicate Records Across Multiple Excel Sheets
A Python utility that uses pandas to find overlapping records across different Excel sheets based on specified key columns.
import pandas as pd
from pathlib import Path
def find_duplicate_records_across_sheets(file_path: str, key_columns: list, sheet_names: list) -> dict:
"""
Detect duplicate records across multiple Excel sheets based on specified key columns.
Args:
file_path: Path to the Excel file
key_co…Create Data Helper Functions in Python for Beginners
Build reusable Python helper functions to load, filter, sort, summarize, and save JSON data — a beginner-friendly starting point for small data pipelines.
import json
from pathlib import Path
from typing import Any, Dict, List
def load_json_file(filepath: str) -> Dict[str, Any]:
"""Load JSON data from a file."""
with Path(filepath).open("r", encoding="utf-8") as file:
return json.load(file)
def filter_by_key(
data: List[Dict[str, Any]], key: str,…ETL in Python: Extract CSV, Transform Dict, Load JSON
Build a simple ETL pipeline in Python that reads a CSV file, transforms each row (stripping whitespace and converting numeric fields), and writes the result to JSON.
import csv
import json
from pathlib import Path
def extract_csv(file_path):
"""Read CSV file and return list of row dictionaries."""
with Path(file_path).open('r', newline='', encoding='utf-8') as f:
reader = csv.DictReader(f)
return list(reader)
def transform_dicts(rows):
"""Transform ro…Generate a Deterministic Hash for Deduplication in Python
Create a stable SHA-256 fingerprint from nested data and file contents to deduplicate records in a data pipeline.
import hashlib
import json
from pathlib import Path
def natural_key_hash(data, salt=""):
"""
Generate a deterministic fingerprint from raw data (dict/list/str).
Uses JSON canonical-ish serialization with sorted keys and SHA-256.
"""
canonical = json.dumps(data, sort_keys=True, separators=(",", ":"…How to Build Data Processing Functions in Python
Create reusable helper functions to load, filter, transform, and aggregate CSV data in Python.
import csv
from pathlib import Path
def load_data(filepath):
"""Load CSV data into a list of dicts."""
with open(filepath, "r", newline="", encoding="utf-8") as f:
return list(csv.DictReader(f))
def filter_rows(rows, column, value):
"""Keep rows where column equals value."""
return [row for…How to Compress Pipeline Output Gzip Per Partition in Python
Compress each partition of pipeline output into a separate gzip file and verify the compressed data by reading it back.
import gzip
import io
import random
from pathlib import Path
def compress_partition(partition_data: list[str], output_path: Path) -> int:
"""Compress a partition of data to a gzip file, returns bytes written."""
with gzip.open(output_path, 'wt', encoding='utf-8') as f:
f.writelines(partition_data)
…How to Count JSON Records in Python
Read a JSON file and count the number of top-level records, handling both list and dictionary structures.
import json
from pathlib import Path
def count_records(json_file):
"""Count top-level records in a JSON file."""
with open(json_file, "r") as f:
data = json.load(f)
# Handle both list of records and dict of records
if isinstance(data, list):
return len(data)
elif isinstance(da…How to Partition Output Files by Date Key in Python
Group output files into a dictionary partitioned by a YYYYMMDD date key extracted from the filename prefix.
from pathlib import Path
from collections import defaultdict
def partition_files_by_date(directory: str) -> dict:
"""Partition output files by date key extracted from filename (YYYYMMDD prefix)."""
path = Path(directory)
partitions = defaultdict(list)
for file in path.iterdir():
if file.i…How to Process CSV Data in Python with a Data Helper
Build a beginner-friendly data helper in Python that loads a CSV file, filters rows by a condition, and summarizes numeric fields.
import csv
from pathlib import Path
DATA = [
{"name": "Alice", "score": 88, "passed": True},
{"name": "Bob", "score": 42, "passed": False},
{"name": "Carol", "score": 95, "passed": True},
]
def load_csv(file_path: Path) -> list[dict]:
with file_path.open(newline="", encoding="utf-8") as f:
r…How to Stream a Large JSONL File Line by Line in Python
Process a large JSON-lines file incrementally using streaming techniques to avoid loading the entire file into memory.
import json
def process_large_file(filepath, chunk_size=8192):
"""
Stream a large JSON-lines file line by line, processing each record
without loading the entire file into memory.
"""
total_count = 0
total_sum = 0
with open(filepath, 'r') as f:
while True:
chunk = …How to Validate Fact Table Grain Row Counts in Python
Validate fact table grain by checking dimension key references, unique grain combinations, duplicate rows, and dimension cardinality from a CSV file.
import csv
import hashlib
from pathlib import Path
def validate_fact_grain(fact_file: Path, expected_dim_keys: dict[str, set[str]]) -> dict:
"""
Validate fact table grain by checking each row's dimension keys exist
in expected dimension tables and row count consistency.
"""
dim_references = {}
…How to create a dated snapshot path for a dataset in Python
Generate a versioned directory path combining a base directory, dataset name, and today's date, ready for creating snapshots in data pipelines.
import datetime
import os
from pathlib import Path
def snapshot_path(base_dir: str, dataset_name: str) -> Path:
"""Return a dated snapshot path for a dataset under a base directory."""
today = datetime.date.today().isoformat()
return Path(base_dir) / dataset_name / today
if __name__ == "__main__":
…Trigger a Pipeline When a New File Appears in a Directory
Poll a directory every 0.5 seconds and return the name of the first new file that appears, or None after a timeout.
import time
from pathlib import Path
def watch_for_file(directory: str, interval: float = 0.5, timeout: float = 10.0) -> str | None:
"""Poll a directory and trigger when a new file appears."""
watch_dir = Path(directory)
watch_dir.mkdir(exist_ok=True)
known_files = set(watch_dir.iterdir())
s…Browse by section
Each section groups closely related Python snippets.
Data pipelines & processing — Python code examples
What you will find here
This page collects data pipelines & processing snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.