Reference library

Data pipelines & processing

ETL-style flows, batch transforms, validation, and moving data between formats.

3 matches

How to Build a Simple Data Pipeline in Python

A beginner-friendly data pipeline that loads JSON, filters records by a field value, and aggregates counts per category.

pipeline json aggregation

Python

import json
from pathlib import Path


def load_json(filepath: str | Path) -> list[dict]:
    """Load a JSON file containing a list of records."""
    with Path(filepath).open("r", encoding="utf-8") as f:
        return json.load(f)


def filter_records(records: list[dict], field: str, value) -> list[dict]:
    """Kee…

10 0 Open

Data pipelines & processing easy

How to Filter Data in Python

Filter a list of dictionaries by exact key-value matches or numerical ranges using concise list comprehensions.

filtering list-comprehension dictionaries

Python

from typing import List, Dict, Any


def filter_data(
    data: List[Dict[str, Any]], key: str, value: Any
) -> List[Dict[str, Any]]:
    """Return records where data[key] equals value."""
    return [record for record in data if record.get(key) == value]


def filter_by_range(
    data: List[Dict[str, Any]], key: str…

12 0 Open

Data pipelines & processing easy

How to Merge Multiple Data Sources in Python

A beginner-friendly helper that merges lists of dictionaries from multiple sources into one combined list using key filtering.

merge pipelines dicts

Python

import json

def merge_pipeline_data(*data_sources, keys=()):
    """Merge multiple data sources (list of dicts) into a single list of merged dicts.
    
    Args:
        *data_sources: One or more lists of dictionaries.
        keys: Tuple of keys to include from each source (empty means all keys).
    Returns:
    …

14 0 Open

Browse by section

Each section groups closely related Python snippets.