Reference library

Streaming & messaging

Kafka-style pub/sub, event consumers, async pipelines, and message-driven workflows.

4 matches

Dedupe processed message IDs in Python

Filters an inbox of messages by removing items whose IDs have already been processed, using a set for fast lookups.

from pathlib import Path
import json


def dedupe_processed_ids(inbox_file: Path, processed_file: Path) -> list:
    processed = set(json.loads(processed_file.read_text()))
    inbox = json.loads(inbox_file.read_text())
    deduped = [item for item in inbox if item["id"] not in processed]
    return deduped


if __nam…

13 0 Open

Streaming & messaging easy

How to Build a Materialized View Updater Consumer Mock in Python

A mock consumer that queues change events and triggers refresh callbacks to simulate materialized view updates.

dataclasses deque mocking

Python

import time
from collections import deque
from dataclasses import dataclass, field
from typing import Callable, Deque, Optional


@dataclass
class MaterializedViewUpdater:
    """Mock updater that consumes change events and refreshes a view."""
    refresh: Optional[Callable[[str], None]] = None
    queue: Deque[tuple…

14 0 Open

Streaming & messaging easy

How to Implement a Priority Queue for Messages in Python

Build a message priority queue with heapq and dataclasses that pops messages by priority, using sequence numbers to keep insertion order.

priority-queue heapq dataclass

Python

import heapq
from dataclasses import dataclass, field
from typing import Any

@dataclass(order=True)
class Message:
    priority: int
    sequence: int = field(compare=False)
    content: str = field(compare=False)

class PriorityQueue:
    def __init__(self):
        self._heap = []

    def push(self, priority: int,…

15 0 Open

Streaming & messaging easy

How to Wrap Message Attributes in a CloudEvent with Python

Create a minimal CloudEvent dataclass that wraps arbitrary message attributes into a JSON envelope, matching CloudEvents 1.0 spec.

cloudevents messaging dataclasses

Python

import json
from dataclasses import dataclass, field, asdict
from typing import Any, Dict
from datetime import datetime, timezone


@dataclass
class CloudEvent:
    message_attributes: Dict[str, Any] = field(default_factory=dict)

    def wrap(self, event_id: str, source: str, event_type: str, data: Any):
        self…

13 0 Open

Browse by section

Each section groups closely related Python snippets.