Reference library

Data pipelines & processing

ETL-style flows, batch transforms, validation, and moving data between formats.

3 matches

ETL in Python: Extract CSV, Transform Dicts, Load JSON

Build a simple ETL pipeline that reads a CSV, normalizes keys and converts price to float, then writes structured JSON.

import csv
import json
from pathlib import Path

def etl_csv_to_json(csv_path: str, json_path: str) -> None:
    """Extract CSV, transform rows to dicts, load to JSON."""
    with open(csv_path, mode='r', newline='', encoding='utf-8') as f:
        reader = csv.DictReader(f)
        records = list(reader)

    # Trans…

12 0 Open

Data pipelines & processing easy

How to Clean and Format Data in Python

This code loads JSON data, cleans records by removing empty fields and normalizing text, then summarizes the results with counts and unique keys.

json data cleaning data pipelines

Python

import json
from pathlib import Path


def load_data(filepath: str) -> dict:
    """Load JSON data from a file."""
    with Path(filepath).open("r", encoding="utf-8") as f:
        return json.load(f)


def clean_records(records: list[dict]) -> list[dict]:
    """Remove empty fields and normalize text to lowercase."""…

13 0 Open

Data pipelines & processing easy

Test a Python Pipeline with Fixture Sample Rows

Test pipeline functions with sample rows provided by a pytest fixture, verifying required keys and value constraints.

pytest fixtures data-pipelines

Python

import pytest


def get_value(data: dict, key: str):
    return data.get(key)


def sample_rows():
    return [
        {"name": "Alice", "age": 30, "city": "London"},
        {"name": "Bob", "age": 25, "city": "Paris"},
        {"name": "Charlie", "age": 35, "city": "Berlin"},
    ]


@pytest.fixture
def sample_data(…

16 0 Open

Browse by section

Each section groups closely related Python snippets.