Reference library

Data pipelines & processing

ETL-style flows, batch transforms, validation, and moving data between formats.

3 matches

ETL in Python: Extract CSV, Transform Dict, Load JSON

Build a simple ETL pipeline in Python that reads a CSV file, transforms each row (stripping whitespace and converting numeric fields), and writes the result to JSON.

etl csv json

Python

import csv
import json
from pathlib import Path

def extract_csv(file_path):
    """Read CSV file and return list of row dictionaries."""
    with Path(file_path).open('r', newline='', encoding='utf-8') as f:
        reader = csv.DictReader(f)
        return list(reader)

def transform_dicts(rows):
    """Transform ro…

14 0 Open

Data pipelines & processing easy

How to Build Data Processing Functions in Python

Create reusable helper functions to load, filter, transform, and aggregate CSV data in Python.

csv pipeline etl

Python

import csv
from pathlib import Path


def load_data(filepath):
    """Load CSV data into a list of dicts."""
    with open(filepath, "r", newline="", encoding="utf-8") as f:
        return list(csv.DictReader(f))


def filter_rows(rows, column, value):
    """Keep rows where column equals value."""
    return [row for…

12 0 Open

Data pipelines & processing medium

How to Stream a Large JSONL File Line by Line in Python

Process a large JSON-lines file incrementally using streaming techniques to avoid loading the entire file into memory.

streaming jsonl large-files

Python

import json

def process_large_file(filepath, chunk_size=8192):
    """
    Stream a large JSON-lines file line by line, processing each record
    without loading the entire file into memory.
    """
    total_count = 0
    total_sum = 0
    
    with open(filepath, 'r') as f:
        while True:
            chunk = …

13 0 Open

Browse by section

Each section groups closely related Python snippets.