How to route late-arriving data to a side output in Python

Separate late-arriving events from a streaming data batch into a dead-letter side output list using a timestamp threshold.

Easy Python 3.9+ Aug 9, 2026 Data pipelines & processing 12 views 0 copies

Python code

36 lines
Python 3.9+
from collections import defaultdict

def late_arriving_side_output(events, late_threshold_ts):
    """
    Mock a streaming pipeline that separates late-arriving data events
    into a side output list (e.g., for dead-letter analysis).

    events: list of (timestamp, data) tuples, timestamps as ints.
    late_threshold_ts: int — events with timestamp < this are considered late.
    Returns: (main_output, side_output_list) — two lists.
    """
    main_output = []
    side_output_list = []

    for ts, data in events:
        if ts < late_threshold_ts:
            side_output_list.append((ts, data))
        else:
            main_output.append((ts, data))

    return main_output, side_output_list


if __name__ == "__main__":
    events = [
        (100, "on-time-1"),
        (95, "late-1"),
        (120, "on-time-2"),
        (90, "late-2"),
        (130, "on-time-3"),
    ]
    threshold = 100

    main, side = late_arriving_side_output(events, threshold)
    print("Main output:", main)
    print("Side output list:", side)

Output

stdout
Main output: [(100, 'on-time-1'), (120, 'on-time-2'), (130, 'on-time-3')]
Side output list: [(95, 'late-1'), (90, 'late-2')]

How it works

The function iterates over timestamped events and compares each timestamp against a late threshold. Events with a timestamp lower than the threshold are pushed to a side output list for dead-letter analysis, while the rest flow to the main output. This simple branching pattern mirrors how streaming frameworks like Apache Flink or Kafka Streams handle late data via side outputs. Because the logic is pure and deterministic, it is easy to unit-test in isolation before integrating into a larger pipeline.

Common mistakes

  • Using <= instead of < for the late threshold, causing threshold-boundary events to be misrouted.
  • Mutating the input events list while iterating, which can cause skipped items.
  • Assuming timestamps are sorted, which breaks the threshold comparison for unsorted batches.

Variations

  1. Return a tuple of (main_list, side_list) as shown, or use a dict keyed by output type for more than two channels.
  2. Use a list comprehension with a partition function from itertools (e.g., tee + filter) for a more functional style.

Real-world use cases

  • Routing out-of-order or delayed events in a real-time analytics pipeline to a dead-letter queue for reprocessing.
  • Separating late session events in clickstream analysis so dashboards only show current data.
  • Isolating stale sensor readings in an IoT ingestion job for anomaly review before archiving.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Data pipelines & processing

Related tutorials and quizzes for this topic.