Reference library

Data pipelines & processing

ETL-style flows, batch transforms, validation, and moving data between formats.

3 matches

Filter Records by Required Fields in Python

Filter a list of dictionaries, keeping only records where every required field is present and not None.

def filter_records(records, required_fields):
    """Return only records that have all required fields non-null."""
    return [
        record for record in records
        if all(record.get(field) is not None for field in required_fields)
    ]


if __name__ == "__main__":
    sample_records = [
        {"name": "Al…

14 0 Open

Data pipelines & processing easy

How to Clean and Format Data in Python

This code loads JSON data, cleans records by removing empty fields and normalizing text, then summarizes the results with counts and unique keys.

json data cleaning data pipelines

Python

import json
from pathlib import Path


def load_data(filepath: str) -> dict:
    """Load JSON data from a file."""
    with Path(filepath).open("r", encoding="utf-8") as f:
        return json.load(f)


def clean_records(records: list[dict]) -> list[dict]:
    """Remove empty fields and normalize text to lowercase."""…

14 0 Open

Data pipelines & processing easy

How to Safely Coerce Strings to Numbers in Python

A safe conversion function that turns strings into integers or floats, returning a fallback value when conversion fails.

type-conversion robust-parsing data-cleaning

Python

import math

def to_number(value, fallback=None):
    """Safely coerce a string to int or float, returning fallback on failure."""
    if isinstance(value, (int, float)):
        return value
    try:
        # Try int first for clean whole numbers
        return int(value)
    except (ValueError, TypeError):
        …

12 0 Open

Browse by section

Each section groups closely related Python snippets.