Data pipelines & processing
ETL-style flows, batch transforms, validation, and moving data between formats.
Build a Python Utility That Detects Duplicate Records Across Multiple Excel Sheets
A Python utility that uses pandas to find overlapping records across different Excel sheets based on specified key columns.
import pandas as pd
from pathlib import Path
def find_duplicate_records_across_sheets(file_path: str, key_columns: list, sheet_names: list) -> dict:
"""
Detect duplicate records across multiple Excel sheets based on specified key columns.
Args:
file_path: Path to the Excel file
key_co…Check Null Rate Threshold in PySpark DataFrame
This PySpark code checks the null rate of specified DataFrame columns against a threshold and returns violations.
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, sum, count
def check_null_rate(df, threshold=0.2, columns=None):
"""
Check null rate for specified columns (or all) against a threshold.
Returns columns that exceed the threshold.
"""
cols = columns or df.columns
total…Deduplicate events by ID within a window in Python
Deduplicate event streams by ID within sliding time windows, keeping the newest occurrence per window using heaps and sets.
import heapq
from collections import defaultdict
def deduplicate_events(events, window_size):
"""Return events deduplicated by id, keeping newest within each sliding window."""
# Index events by (timestamp, id) for deterministic ordering
events_by_id = defaultdict(list)
for ts, eid, *payload in events…Enrich a stream with reference data by key lookup in Python
Uses streamz to join each incoming record to a reference dictionary by name, adding department and level fields or defaults.
from streamz import Stream
reference = {"alice": {"dept": "eng", "level": 3}, "bob": {"dept": "sales", "level": 5}}
def enrich(record):
name = record.get("name")
ref = reference.get(name)
joined = dict(record)
if ref:
joined.update(ref)
else:
joined["dept"] = "unknown"
joi…Extract Schema.org Structured Data from Any Website in Python
A Python tool that fetches a webpage and extracts all JSON-LD structured data (Schema.org) embedded in <script> tags with type="application/ld+json".
import requests
from bs4 import BeautifulSoup
import json
def extract_schema_org(url):
"""Extract structured data (Schema.org) from a website."""
try:
response = requests.get(url, timeout=10)
response.raise_for_status()
except requests.exceptions.RequestException as e:
return {"err…How to Count Events by Minute with a Tumbling Window in Python
Group timestamps into fixed 60-second tumbling windows and count events per bucket using a dict.
from collections import defaultdict
from datetime import datetime, timedelta
def tumbling_window_count(events, window_seconds=60):
buckets = defaultdict(int)
for event in events:
ts = datetime.fromisoformat(event["timestamp"])
bucket_start = ts - timedelta(seconds=ts.second % window_seconds,
…How to Find Missing Values in Large Datasets in Python
Analyze missing values across multiple large pandas DataFrames with counts and percentages.
import pandas as pd
import numpy as np
def find_missing_values_summary(datasets):
"""Analyze missing values across multiple datasets (dict of name: DataFrame)."""
summary = {}
for name, df in datasets.items():
missing_count = df.isnull().sum()
total_rows = len(df)
missing_pct = (mi…How to Implement SCD Type 1 Overwrite in Python with SQLite
Implement SCD Type 1 dimension updates in Python using SQLite — overwrite existing rows with new data while preserving keys.
import sqlite3
# Simulate a dimension table with SCD Type 1 (overwrite)
conn = sqlite3.connect(":memory:")
cursor = conn.cursor()
# Create dimension table
cursor.execute("""
CREATE TABLE customer_dim (
customer_id INTEGER PRIMARY KEY,
customer_name TEXT,
city TEXT,
updated_at TEXT…How to Implement Slowly Changing Dimension Type 2 History in Python
Build a type-2 slowly changing dimension pipeline that closes old records and opens new ones when customer data changes.
from datetime import datetime, timedelta
def apply_scd_type2(records, current_date):
"""Returns active records after inserting new records with type-2 history."""
history = []
active = {}
for record in records:
key = record["customer_id"]
if key in active:
active[key]["end…How to Stream a Large JSONL File Line by Line in Python
Process a large JSON-lines file incrementally using streaming techniques to avoid loading the entire file into memory.
import json
def process_large_file(filepath, chunk_size=8192):
"""
Stream a large JSON-lines file line by line, processing each record
without loading the entire file into memory.
"""
total_count = 0
total_sum = 0
with open(filepath, 'r') as f:
while True:
chunk = …How to Topologically Sort a DAG in Python
Compute a valid execution order for tasks with dependencies using Kahn's algorithm in Python.
from collections import defaultdict, deque
def topological_order(dependencies):
graph = defaultdict(list)
in_degree = defaultdict(int)
tasks = set(dependencies.keys())
for task, depends_on in dependencies.items():
for d in depends_on:
graph[d].append(task)
in_degree[t…How to Validate Fact Table Grain Row Counts in Python
Validate fact table grain by checking dimension key references, unique grain combinations, duplicate rows, and dimension cardinality from a CSV file.
import csv
import hashlib
from pathlib import Path
def validate_fact_grain(fact_file: Path, expected_dim_keys: dict[str, set[str]]) -> dict:
"""
Validate fact table grain by checking each row's dimension keys exist
in expected dimension tables and row count consistency.
"""
dim_references = {}
…How to perform a star schema join in Python
Denormalize mock fact and dimension tables by building lookup dicts and enriching each sales fact with customer, product, and date attributes.
from datetime import date
# Mock dimension tables
customers = [
{"customer_id": 1, "name": "Alice", "city": "New York"},
{"customer_id": 2, "name": "Bob", "city": "Los Angeles"},
{"customer_id": 3, "name": "Carol", "city": "Chicago"},
]
products = [
{"product_id": 101, "name": "Laptop", "category": "…Implement an Out-of-Order Sort Buffer with a Heap in Python
Buffers out-of-order indices from a stream and emits them in sorted order using a min-heap with a sliding window.
import heapq
from collections import deque
class OutOfOrderSorter:
def __init__(self, buffer_size):
self.buffer_size = buffer_size
self.buffer = deque(maxlen=buffer_size)
self.heap = []
self.next_expected_index = 0
self.result = []
def push(self, item):
heapq.…Map Partition Over Chunks in Python with Multiprocessing and Mock
Process data in chunks across multiple CPU cores using multiprocessing Pool.map, and mock the chunk function to test partitioning behavior without heavy computation.
from multiprocessing import Pool
from unittest.mock import patch, Mock
def process_chunk(chunk):
return [x * x for x in chunk]
def map_partition_over_chunks(data, chunk_size, process_func=process_chunk):
chunks = [data[i:i + chunk_size] for i in range(0, len(data), chunk_size)]
with Pool() as pool:
…Normalize Timestamps to UTC DateTime in Python
Convert timestamps in multiple formats to UTC-aware datetime objects using datetime.strptime and astimezone.
from datetime import datetime, timezone
raw_timestamps = [
"2024-01-15 14:30:00+02:00",
"17/05/2024 09:15:00 -0500",
"2024-03-01T22:45:00Z",
"2024-06-20 08:00:00+09:30"
]
def parse_and_convert(ts: str) -> datetime:
normalized_ts = ts.strip().replace("Z", "+00:00")
formats = [
"%Y-%m-%…Pivot long to wide transformation dict
Transform a list of dictionaries from long format to wide format by pivoting on a key column and aggregating values, using pure Python.
def pivot_long_to_wide(rows, key_col, value_col, id_cols=None):
"""
Convert long-format data (list of dicts) to wide format.
Args:
rows: List of dicts in long format
key_col: Column name to pivot on (becomes new column headers)
value_col: Column name whose values become the cel…Python Exponential Backoff Retry Example
Retry a flaky function with exponential backoff and jitter-free delays, printing each attempt and finally returning the successful result.
import random
import time
def flaky_function():
if random.random() < 0.6:
raise ConnectionError("Temporary network error")
return "success"
def retry_with_exponential_backoff(func, max_retries=5, base_delay=1.0):
for attempt in range(max_retries + 1):
try:
return func()
…Browse by section
Each section groups closely related Python snippets.
Data pipelines & processing — Python code examples
What you will find here
This page collects data pipelines & processing snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.