Reference library

Data pipelines & processing

ETL-style flows, batch transforms, validation, and moving data between formats.

3 matches

Generate a Deterministic Hash for Deduplication in Python

Create a stable SHA-256 fingerprint from nested data and file contents to deduplicate records in a data pipeline.

import hashlib
import json
from pathlib import Path

def natural_key_hash(data, salt=""):
    """
    Generate a deterministic fingerprint from raw data (dict/list/str).
    Uses JSON canonical-ish serialization with sorted keys and SHA-256.
    """
    canonical = json.dumps(data, sort_keys=True, separators=(",", ":"…

14 0 Open

Data pipelines & processing easy

How to Hash Email Addresses in a PII Masking Pipeline in Python

Replaces every email address in a text string with its SHA-256 hash to protect personally identifiable information (PII).

pii hashing sha256

Python

import hashlib
import re

def hash_email(email: str) -> str:
    """Mask an email address by hashing it with SHA-256."""
    normalized = email.strip().lower()
    return hashlib.sha256(normalized.encode("utf-8")).hexdigest()

def mask_pii_emails(text: str) -> str:
    """Replace all email addresses in text with their…

14 0 Open

Data pipelines & processing easy

How to shard output by primary key hash mod N in Python

This code computes a consistent shard index for any primary key string using an MD5 hash mod the number of shards, enabling stable key-based data distribution.

hashing sharding hashlib

Python

import hashlib

def shard_id(primary_key: str, num_shards: int) -> int:
    """Return the shard index for a primary key using MD5 hash mod N."""
    digest = hashlib.md5(primary_key.encode("utf-8")).hexdigest()
    hash_int = int(digest, 16)
    return hash_int % num_shards

if __name__ == "__main__":
    keys = ["use…

11 0 Open

Browse by section

Each section groups closely related Python snippets.