Reference library

Data pipelines & processing

ETL-style flows, batch transforms, validation, and moving data between formats.

3 matches

How to Convert Data Types in a Python Data Pipeline

Demonstrates a simple Python data pipeline that converts string values to proper types (bool, int, float, datetime) and outputs structured JSON.

data-pipeline type-conversion json

Python

import json
from datetime import datetime

def convert_value(value):
    """Convert string values to appropriate Python types."""
    if value.lower() == "true":
        return True
    if value.lower() == "false":
        return False
    if value.isdigit():
        return int(value)
    try:
        return float(val…

11 0 Open

Data pipelines & processing easy

How to Validate Data in a Python Pipeline

A helper module to validate common record types — email, positive integer, and non-empty string list — before processing data in a pipeline.

data-validation pipelines type-checking

Python

from typing import Any, Iterable


def is_valid_email(email: str) -> bool:
    """Basic email check: one '@', no spaces, dot after '@'."""
    if "@" not in email or " " in email:
        return False
    local, _, domain = email.partition("@")
    return bool(local) and "." in domain


def is_positive_int(value: Any)…

12 0 Open

Data pipelines & processing easy

Validate dict schema at pipeline boundary in Python

This code validates a dictionary against a TypedDict schema at a pipeline boundary, enforcing required fields and types with custom error messages.

validation dict typeddict

Python

from typing import Any, TypedDict


class Person(TypedDict):
    name: str
    age: int
    email: str


def validate_person(data: dict[str, Any]) -> Person:
    errors: list[str] = []

    if not isinstance(data.get("name"), str) or not data["name"].strip():
        errors.append("name must be a non-empty string")
  …

13 0 Open

Browse by section

Each section groups closely related Python snippets.