Reference library

Data pipelines & processing

ETL-style flows, batch transforms, validation, and moving data between formats.

6 matches

How to Clean and Format Data in Python

This code loads JSON data, cleans records by removing empty fields and normalizing text, then summarizes the results with counts and unique keys.

json data cleaning data pipelines

Python

import json
from pathlib import Path


def load_data(filepath: str) -> dict:
    """Load JSON data from a file."""
    with Path(filepath).open("r", encoding="utf-8") as f:
        return json.load(f)


def clean_records(records: list[dict]) -> list[dict]:
    """Remove empty fields and normalize text to lowercase."""…

13 0 Open

Data pipelines & processing easy

How to Explode an Array Field into Multiple Rows in Python

This code flattens a list of dictionaries by exploding each array field value into its own row, duplicating the other fields as needed.

data transformation arrays flattening

Python

from collections import defaultdict

data = [
    {"id": 1, "name": "Alice", "tags": ["python", "data", "ai"]},
    {"id": 2, "name": "Bob", "tags": ["web", "devops"]},
    {"id": 3, "name": "Carol", "tags": []},
]

def explode_array_field(records, array_field):
    result = []
    for record in records:
        for v…

11 0 Open

Data pipelines & processing easy

How to Hash Email Addresses in a PII Masking Pipeline in Python

Replaces every email address in a text string with its SHA-256 hash to protect personally identifiable information (PII).

pii hashing sha256

Python

import hashlib
import re

def hash_email(email: str) -> str:
    """Mask an email address by hashing it with SHA-256."""
    normalized = email.strip().lower()
    return hashlib.sha256(normalized.encode("utf-8")).hexdigest()

def mask_pii_emails(text: str) -> str:
    """Replace all email addresses in text with their…

14 0 Open

Data pipelines & processing easy

How to Unpivot Wide to Long with pandas melt in Python

This code demonstrates how to use pandas.melt to unpivot a wide DataFrame into a tidy long format, converting subject columns into rows.

pandas melt reshape

Python

import pandas as pd

# Sample wide-format data
df_wide = pd.DataFrame({
    'id': [1, 2, 3],
    'name': ['Alice', 'Bob', 'Charlie'],
    'math': [90, 85, 95],
    'science': [80, 92, 88]
})

print("Original wide DataFrame:")
print(df_wide)

# Melt: unpivot subject columns into rows
df_long = pd.melt(
    df_wide,
   …

15 0 Open

Data pipelines & processing medium

Normalize Timestamps to UTC DateTime in Python

Convert timestamps in multiple formats to UTC-aware datetime objects using datetime.strptime and astimezone.

datetime timezone utc

Python

from datetime import datetime, timezone

raw_timestamps = [
    "2024-01-15 14:30:00+02:00",
    "17/05/2024 09:15:00 -0500",
    "2024-03-01T22:45:00Z",
    "2024-06-20 08:00:00+09:30"
]

def parse_and_convert(ts: str) -> datetime:
    normalized_ts = ts.strip().replace("Z", "+00:00")
    formats = [
        "%Y-%m-%…

14 0 Open

Data pipelines & processing medium

Pivot long to wide transformation dict

Transform a list of dictionaries from long format to wide format by pivoting on a key column and aggregating values, using pure Python.

pivot transformation data-cleaning

Python

def pivot_long_to_wide(rows, key_col, value_col, id_cols=None):
    """
    Convert long-format data (list of dicts) to wide format.
    
    Args:
        rows: List of dicts in long format
        key_col: Column name to pivot on (becomes new column headers)
        value_col: Column name whose values become the cel…

11 0 Open

Browse by section

Each section groups closely related Python snippets.