Reference library

Data pipelines & processing

ETL-style flows, batch transforms, validation, and moving data between formats.

4 matches

Extract Schema.org Structured Data from Any Website in Python

A Python tool that fetches a webpage and extracts all JSON-LD structured data (Schema.org) embedded in <script> tags with type="application/ld+json".

web-scraping structured-data schema-org

Python

import requests
from bs4 import BeautifulSoup
import json

def extract_schema_org(url):
    """Extract structured data (Schema.org) from a website."""
    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()
    except requests.exceptions.RequestException as e:
        return {"err…

51 0 Open

Data pipelines & processing medium

How to Validate Fact Table Grain Row Counts in Python

Validate fact table grain by checking dimension key references, unique grain combinations, duplicate rows, and dimension cardinality from a CSV file.

csv data validation etl

Python

import csv
import hashlib
from pathlib import Path


def validate_fact_grain(fact_file: Path, expected_dim_keys: dict[str, set[str]]) -> dict:
    """
    Validate fact table grain by checking each row's dimension keys exist
    in expected dimension tables and row count consistency.
    """
    dim_references = {}
  …

13 0 Open

Data pipelines & processing medium

Implement an Out-of-Order Sort Buffer with a Heap in Python

Buffers out-of-order indices from a stream and emits them in sorted order using a min-heap with a sliding window.

heapq sorting streaming

Python

import heapq
from collections import deque


class OutOfOrderSorter:
    def __init__(self, buffer_size):
        self.buffer_size = buffer_size
        self.buffer = deque(maxlen=buffer_size)
        self.heap = []
        self.next_expected_index = 0
        self.result = []

    def push(self, item):
        heapq.…

12 0 Open

Data pipelines & processing medium

Pivot long to wide transformation dict

Transform a list of dictionaries from long format to wide format by pivoting on a key column and aggregating values, using pure Python.

pivot transformation data-cleaning

Python

def pivot_long_to_wide(rows, key_col, value_col, id_cols=None):
    """
    Convert long-format data (list of dicts) to wide format.
    
    Args:
        rows: List of dicts in long format
        key_col: Column name to pivot on (becomes new column headers)
        value_col: Column name whose values become the cel…

11 0 Open

Browse by section

Each section groups closely related Python snippets.