Data pipelines & processing
ETL-style flows, batch transforms, validation, and moving data between formats.
Attach Source File Metadata to Records in Python
Add a source filename field to each record in a list by merging a new key into every dictionary using a dict unpacking comprehension.
from pathlib import Path
import json
def attach_source_metadata(records, source_file):
"""Attach source filename metadata to each record."""
return [
{**record, "source": Path(source_file).name}
for record in records
]
if __name__ == "__main__":
source = "/data/raw/customers.csv"
…Count Records Processed per Category in Python
Use a Counter dictionary to track how many records of each type (ok, error, retry) were processed in a data pipeline.
from collections import Counter
import random
processed_counter = Counter()
def process_records(records):
for record in records:
processed_counter[record] += 1
return len(records)
if __name__ == "__main__":
sample_records = [random.choice(["ok", "error", "retry"]) for _ in range(10)]
print(f…Deduplicate events by ID within a window in Python
Deduplicate event streams by ID within sliding time windows, keeping the newest occurrence per window using heaps and sets.
import heapq
from collections import defaultdict
def deduplicate_events(events, window_size):
"""Return events deduplicated by id, keeping newest within each sliding window."""
# Index events by (timestamp, id) for deterministic ordering
events_by_id = defaultdict(list)
for ts, eid, *payload in events…ETL in Python: Extract CSV, Transform Dict, Load JSON
Build a simple ETL pipeline in Python that reads a CSV file, transforms each row (stripping whitespace and converting numeric fields), and writes the result to JSON.
import csv
import json
from pathlib import Path
def extract_csv(file_path):
"""Read CSV file and return list of row dictionaries."""
with Path(file_path).open('r', newline='', encoding='utf-8') as f:
reader = csv.DictReader(f)
return list(reader)
def transform_dicts(rows):
"""Transform ro…ETL in Python: Extract CSV, Transform Dicts, Load JSON
Build a simple ETL pipeline that reads a CSV, normalizes keys and converts price to float, then writes structured JSON.
import csv
import json
from pathlib import Path
def etl_csv_to_json(csv_path: str, json_path: str) -> None:
"""Extract CSV, transform rows to dicts, load to JSON."""
with open(csv_path, mode='r', newline='', encoding='utf-8') as f:
reader = csv.DictReader(f)
records = list(reader)
# Trans…Enrich Events with Geo IP Data in Python
Returns a copy of each event dictionary, enriched with a geo-location dict from a mock IP-to-geo lookup table, with a fallback for unknown IPs.
import ipaddress
GEO_IP_DB = {
"192.168.1.10": {"country": "US", "city": "New York", "lat": 40.7128, "lon": -74.0060},
"10.0.0.5": {"country": "DE", "city": "Berlin", "lat": 52.5200, "lon": 13.4050},
"172.16.0.8": {"country": "JP", "city": "Tokyo", "lat": 35.6762, "lon": 139.6503},
}
EVENTS = [
{"id…Enrich a stream with reference data by key lookup in Python
Uses streamz to join each incoming record to a reference dictionary by name, adding department and level fields or defaults.
from streamz import Stream
reference = {"alice": {"dept": "eng", "level": 3}, "bob": {"dept": "sales", "level": 5}}
def enrich(record):
name = record.get("name")
ref = reference.get(name)
joined = dict(record)
if ref:
joined.update(ref)
else:
joined["dept"] = "unknown"
joi…Fan Out Records to Multiple Sinks in Python
Distribute the same records across multiple target sinks (database, API, queue, etc.) using a defaultdict-based fan-out pattern.
import json
from collections import defaultdict
SINKS = ["database", "api", "message_queue", "data_lake", "monitoring"]
def fan_out(records, *sinks):
dist = defaultdict(list)
for record in records:
for sink in sinks:
dist[sink].append(record)
return dict(dist)
if __name__ == "__main_…Filter Records by Required Fields in Python
Filter a list of dictionaries, keeping only records where every required field is present and not None.
def filter_records(records, required_fields):
"""Return only records that have all required fields non-null."""
return [
record for record in records
if all(record.get(field) is not None for field in required_fields)
]
if __name__ == "__main__":
sample_records = [
{"name": "Al…How to Count Events by Minute with a Tumbling Window in Python
Group timestamps into fixed 60-second tumbling windows and count events per bucket using a dict.
from collections import defaultdict
from datetime import datetime, timedelta
def tumbling_window_count(events, window_seconds=60):
buckets = defaultdict(int)
for event in events:
ts = datetime.fromisoformat(event["timestamp"])
bucket_start = ts - timedelta(seconds=ts.second % window_seconds,
…How to Count JSON Records in Python
Read a JSON file and count the number of top-level records, handling both list and dictionary structures.
import json
from pathlib import Path
def count_records(json_file):
"""Count top-level records in a JSON file."""
with open(json_file, "r") as f:
data = json.load(f)
# Handle both list of records and dict of records
if isinstance(data, list):
return len(data)
elif isinstance(da…How to Explode an Array Field into Multiple Rows in Python
This code flattens a list of dictionaries by exploding each array field value into its own row, duplicating the other fields as needed.
from collections import defaultdict
data = [
{"id": 1, "name": "Alice", "tags": ["python", "data", "ai"]},
{"id": 2, "name": "Bob", "tags": ["web", "devops"]},
{"id": 3, "name": "Carol", "tags": []},
]
def explode_array_field(records, array_field):
result = []
for record in records:
for v…How to Filter Data in Python
Filter a list of dictionaries by exact key-value matches or numerical ranges using concise list comprehensions.
from typing import List, Dict, Any
def filter_data(
data: List[Dict[str, Any]], key: str, value: Any
) -> List[Dict[str, Any]]:
"""Return records where data[key] equals value."""
return [record for record in data if record.get(key) == value]
def filter_by_range(
data: List[Dict[str, Any]], key: str…How to Group Data by Key in Python
Group a list of dictionaries by a specified key using a defaultdict and compute per-group averages.
from collections import defaultdict
def group_by_key(data, key):
grouped = defaultdict(list)
for item in data:
grouped[item[key]].append(item)
return dict(grouped)
if __name__ == "__main__":
records = [
{"name": "Alice", "dept": "Engineering", "score": 85},
{"name": "Bob", "de…How to Group Rows by Key into Nested Arrays in Python
This code groups rows in a list of dictionaries by a specified key and returns a dictionary with each key mapped to a list of values from another key.
from collections import defaultdict
def implode_rows(rows, key, value_key):
grouped = defaultdict(list)
for row in rows:
grouped[row[key]].append(row[value_key])
return dict(grouped)
if __name__ == "__main__":
data = [
{"category": "fruit", "item": "apple"},
{"category": "fr…How to Merge Incremental Snapshot Upsert Dict in Python
Merge a snapshot dict into a base dict, recursively updating nested dictionaries while preferring snapshot values on conflicts.
def merge_upsert(base: dict, snapshot: dict) -> dict:
"""
Merge a snapshot dict into a base dict, preferring snapshot values
on key conflicts (upsert semantics). Nested dicts are merged recursively.
"""
result = dict(base)
for key, value in snapshot.items():
if key in result and i…How to Merge Multiple Data Sources in Python
A beginner-friendly helper that merges lists of dictionaries from multiple sources into one combined list using key filtering.
import json
def merge_pipeline_data(*data_sources, keys=()):
"""Merge multiple data sources (list of dicts) into a single list of merged dicts.
Args:
*data_sources: One or more lists of dictionaries.
keys: Tuple of keys to include from each source (empty means all keys).
Returns:
…How to Partition Output Files by Date Key in Python
Group output files into a dictionary partitioned by a YYYYMMDD date key extracted from the filename prefix.
from pathlib import Path
from collections import defaultdict
def partition_files_by_date(directory: str) -> dict:
"""Partition output files by date key extracted from filename (YYYYMMDD prefix)."""
path = Path(directory)
partitions = defaultdict(list)
for file in path.iterdir():
if file.i…How to Reduce Aggregate Counts from Mapped Chunks in Python
Combine a list of mapped chunk dictionaries into a single aggregated count dictionary using functools.reduce.
from functools import reduce
from collections import defaultdict
def aggregate_chunks(mapped_chunks):
"""Combine mapped chunk counts into a single aggregate dict."""
return reduce(
lambda acc, chunk: {
**acc,
**{k: acc.get(k, 0) + v for k, v in chunk.items()}
},
…How to Sort a List of Dictionaries by Key in Python
A reusable helper function that sorts a list of dictionaries by a specified key, with optional descending order support.
from typing import List
def sort_records(records: List[dict], key: str, descending: bool = False) -> List[dict]:
"""Sort a list of dictionaries by a specified key."""
return sorted(records, key=lambda record: record[key], reverse=descending)
def demonstrate_sorting() -> None:
users = [
{"name": …How to perform a star schema join in Python
Denormalize mock fact and dimension tables by building lookup dicts and enriching each sales fact with customer, product, and date attributes.
from datetime import date
# Mock dimension tables
customers = [
{"customer_id": 1, "name": "Alice", "city": "New York"},
{"customer_id": 2, "name": "Bob", "city": "Los Angeles"},
{"customer_id": 3, "name": "Carol", "city": "Chicago"},
]
products = [
{"product_id": 101, "name": "Laptop", "category": "…Pivot long to wide transformation dict
Transform a list of dictionaries from long format to wide format by pivoting on a key column and aggregating values, using pure Python.
def pivot_long_to_wide(rows, key_col, value_col, id_cols=None):
"""
Convert long-format data (list of dicts) to wide format.
Args:
rows: List of dicts in long format
key_col: Column name to pivot on (becomes new column headers)
value_col: Column name whose values become the cel…Validate dict schema at pipeline boundary in Python
This code validates a dictionary against a TypedDict schema at a pipeline boundary, enforcing required fields and types with custom error messages.
from typing import Any, TypedDict
class Person(TypedDict):
name: str
age: int
email: str
def validate_person(data: dict[str, Any]) -> Person:
errors: list[str] = []
if not isinstance(data.get("name"), str) or not data["name"].strip():
errors.append("name must be a non-empty string")
…Browse by section
Each section groups closely related Python snippets.
Data pipelines & processing — Python code examples
What you will find here
This page collects data pipelines & processing snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.