Data pipelines & processing
ETL-style flows, batch transforms, validation, and moving data between formats.
Generate a Deterministic Hash for Deduplication in Python
Create a stable SHA-256 fingerprint from nested data and file contents to deduplicate records in a data pipeline.
import hashlib
import json
from pathlib import Path
def natural_key_hash(data, salt=""):
"""
Generate a deterministic fingerprint from raw data (dict/list/str).
Uses JSON canonical-ish serialization with sorted keys and SHA-256.
"""
canonical = json.dumps(data, sort_keys=True, separators=(",", ":"…How to Hash Email Addresses in a PII Masking Pipeline in Python
Replaces every email address in a text string with its SHA-256 hash to protect personally identifiable information (PII).
import hashlib
import re
def hash_email(email: str) -> str:
"""Mask an email address by hashing it with SHA-256."""
normalized = email.strip().lower()
return hashlib.sha256(normalized.encode("utf-8")).hexdigest()
def mask_pii_emails(text: str) -> str:
"""Replace all email addresses in text with their…Browse by section
Each section groups closely related Python snippets.
Data pipelines & processing — Python code examples
What you will find here
This page collects data pipelines & processing snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.