Reference library

Data pipelines & processing

ETL-style flows, batch transforms, validation, and moving data between formats.

4 matches

Generate a Mock CDC Changelog in Python

Simulate a CDC changelog with INSERT, UPDATE, and DELETE operations, timestamps, and record snapshots for testing data pipelines.

cdc changelog mock-data

Python

import json
from datetime import datetime, timedelta


def generate_mock_changelog(records, operations=("INSERT", "UPDATE", "DELETE")):
    """Simulate a CDC changelog from a list of record snapshots."""
    base_time = datetime(2025, 1, 1, 8, 0, 0)
    changelog = []
    for idx, record in enumerate(records):
       …

15 0 Open

Data pipelines & processing medium

How to Count Events by Minute with a Tumbling Window in Python

Group timestamps into fixed 60-second tumbling windows and count events per bucket using a dict.

datetime grouping time-window

Python

from collections import defaultdict
from datetime import datetime, timedelta


def tumbling_window_count(events, window_seconds=60):
    buckets = defaultdict(int)
    for event in events:
        ts = datetime.fromisoformat(event["timestamp"])
        bucket_start = ts - timedelta(seconds=ts.second % window_seconds,
…

13 0 Open

Data pipelines & processing easy

How to List Failed Records in a Dead Letter Queue Mock in Python

A mock Dead Letter Queue stores failed processing records with error details and timestamps, lists them, and exports to JSON.

dead-letter-queue json logging

Python

import json
from datetime import datetime, timedelta
import random


class DeadLetterQueue:
    def __init__(self):
        self.failed_records = []

    def add_failed_record(self, record_id, payload, error_message):
        self.failed_records.append({
            "record_id": record_id,
            "payload": paylo…

13 0 Open

Data pipelines & processing medium

Normalize Timestamps to UTC DateTime in Python

Convert timestamps in multiple formats to UTC-aware datetime objects using datetime.strptime and astimezone.

datetime timezone utc

Python

from datetime import datetime, timezone

raw_timestamps = [
    "2024-01-15 14:30:00+02:00",
    "17/05/2024 09:15:00 -0500",
    "2024-03-01T22:45:00Z",
    "2024-06-20 08:00:00+09:30"
]

def parse_and_convert(ts: str) -> datetime:
    normalized_ts = ts.strip().replace("Z", "+00:00")
    formats = [
        "%Y-%m-%…

14 0 Open

Browse by section

Each section groups closely related Python snippets.