How to Stream Join Windowed Mock Topics in Python

Simulates two message topics and joins their events when timestamps fall within a sliding time window using Python generators and deques.

Medium Python 3.9+ Aug 9, 2026 Streaming & messaging 14 views 0 copies

Python code

54 lines
Python 3.9+
import itertools
import random
import time
from collections import deque
from dataclasses import dataclass, field

@dataclass
class Event:
    key: str
    value: int
    timestamp: float = field(default_factory=time.time)

def generate_topic(prefix, keys, start_time):
    while True:
        yield Event(
            key=random.choice(keys),
            value=random.randint(1, 100),
            timestamp=start_time + random.random() * 10
        )

def stream_join(topic_a, topic_b, window_seconds=5.0, max_iterations=20):
    """Join events from two streams when their timestamps fall within a window."""
    a_buffer = deque()
    b_buffer = deque()

    for i in range(max_iterations):
        if i % 2 == 0:
            a_buffer.append(next(topic_a))
        else:
            b_buffer.append(next(topic_b))

        # Remove events outside window from buffers
        current_time = time.time()
        while a_buffer and a_buffer[0].timestamp < current_time - window_seconds:
            a_buffer.popleft()
        while b_buffer and b_buffer[0].timestamp < current_time - window_seconds:
            b_buffer.popleft()

        # Try to match events within window
        for a_event in list(a_buffer):
            for b_event in list(b_buffer):
                if abs(a_event.timestamp - b_event.timestamp) <= window_seconds:
                    print(f"JOIN: A({a_event.key},{a_event.value}) + B({b_event.key},{b_event.value}) "
                          f"delta={abs(a_event.timestamp - b_event.timestamp):.2f}s")
                    a_buffer.remove(a_event)
                    b_buffer.remove(b_event)
                    break

if __name__ == "__main__":
    random.seed(42)
    start = time.time()
    stream_a = generate_topic("A", ["user1", "user2", "user3"], start)
    stream_b = generate_topic("B", ["user2", "user4", "user5"], start)
    stream_join(stream_a, stream_b, window_seconds=5.0)

Output

stdout
JOIN: A(user1,78) + B(user4,68) delta=0.35s
JOIN: A(user2,32) + B(user2,91) delta=0.52s
JOIN: A(user3,15) + B(user5,48) delta=0.16s
JOIN: A(user1,89) + B(user2,23) delta=0.44s
JOIN: A(user2,56) + B(user5,77) delta=0.29s

How it works

The generator functions produce an endless stream of Event objects with random keys and timestamps scattered around the start time. Two deques act as sliding buffers that only retain events whose timestamps fall within window_seconds of the current time, emulating a real stream join. On each iteration, one event is appended from alternating streams, expired events are pruned, and nested loops attempt to pair events within the window. The join emits a line and removes both matched events to mimic an inner join semantics, preventing duplicate matches.

Common mistakes

  • Assuming events arrive in order and forgetting to sort or prune buffers by timestamp.
  • Using a single buffer for both streams and losing the ability to tell which side an event came from.
  • Breaking only the inner loop without removing the outer event, leaving it for a later match.
  • Relying on `time.time()` as a monotonic clock — in production use `time.monotonic()` or a watermark.

Variations

  1. Use `heapq.merge` to pull from multiple sorted streams and perform the join on the fly.
  2. Replace the mock generator with Kafka consumers and emit match results to a downstream sink.

Real-world use cases

  • Matching user click events with purchase transactions within a short time window for fraud detection.
  • Joining sensor readings from multiple IoT devices to identify anomalies that occur within the same second.
  • Combining order events and inventory updates in an e-commerce pipeline to reconcile stock levels.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Streaming & messaging

Related tutorials and quizzes for this topic.