Data pipelines & processing
ETL-style flows, batch transforms, validation, and moving data between formats.
Extract Schema.org Structured Data from Any Website in Python
A Python tool that fetches a webpage and extracts all JSON-LD structured data (Schema.org) embedded in <script> tags with type="application/ld+json".
import requests
from bs4 import BeautifulSoup
import json
def extract_schema_org(url):
"""Extract structured data (Schema.org) from a website."""
try:
response = requests.get(url, timeout=10)
response.raise_for_status()
except requests.exceptions.RequestException as e:
return {"err…How to List Failed Records in a Dead Letter Queue Mock in Python
A mock Dead Letter Queue stores failed processing records with error details and timestamps, lists them, and exports to JSON.
import json
from datetime import datetime, timedelta
import random
class DeadLetterQueue:
def __init__(self):
self.failed_records = []
def add_failed_record(self, record_id, payload, error_message):
self.failed_records.append({
"record_id": record_id,
"payload": paylo…How to Merge Multiple Data Sources in Python
A beginner-friendly helper that merges lists of dictionaries from multiple sources into one combined list using key filtering.
import json
def merge_pipeline_data(*data_sources, keys=()):
"""Merge multiple data sources (list of dicts) into a single list of merged dicts.
Args:
*data_sources: One or more lists of dictionaries.
keys: Tuple of keys to include from each source (empty means all keys).
Returns:
…Browse by section
Each section groups closely related Python snippets.
Data pipelines & processing — Python code examples
What you will find here
This page collects data pipelines & processing snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.