How to route late-arriving data to a side output in Python
Separate late-arriving events from a streaming data batch into a dead-letter side output list using a timestamp threshold.
Python code
36 linesfrom collections import defaultdict
def late_arriving_side_output(events, late_threshold_ts):
"""
Mock a streaming pipeline that separates late-arriving data events
into a side output list (e.g., for dead-letter analysis).
events: list of (timestamp, data) tuples, timestamps as ints.
late_threshold_ts: int — events with timestamp < this are considered late.
Returns: (main_output, side_output_list) — two lists.
"""
main_output = []
side_output_list = []
for ts, data in events:
if ts < late_threshold_ts:
side_output_list.append((ts, data))
else:
main_output.append((ts, data))
return main_output, side_output_list
if __name__ == "__main__":
events = [
(100, "on-time-1"),
(95, "late-1"),
(120, "on-time-2"),
(90, "late-2"),
(130, "on-time-3"),
]
threshold = 100
main, side = late_arriving_side_output(events, threshold)
print("Main output:", main)
print("Side output list:", side)
Output
Main output: [(100, 'on-time-1'), (120, 'on-time-2'), (130, 'on-time-3')]
Side output list: [(95, 'late-1'), (90, 'late-2')]
How it works
The function iterates over timestamped events and compares each timestamp against a late threshold. Events with a timestamp lower than the threshold are pushed to a side output list for dead-letter analysis, while the rest flow to the main output. This simple branching pattern mirrors how streaming frameworks like Apache Flink or Kafka Streams handle late data via side outputs. Because the logic is pure and deterministic, it is easy to unit-test in isolation before integrating into a larger pipeline.
Common mistakes
- Using <= instead of < for the late threshold, causing threshold-boundary events to be misrouted.
- Mutating the input events list while iterating, which can cause skipped items.
- Assuming timestamps are sorted, which breaks the threshold comparison for unsorted batches.
Variations
- Return a tuple of (main_list, side_list) as shown, or use a dict keyed by output type for more than two channels.
- Use a list comprehension with a partition function from itertools (e.g., tee + filter) for a more functional style.
Real-world use cases
- Routing out-of-order or delayed events in a real-time analytics pipeline to a dead-letter queue for reprocessing.
- Separating late session events in clickstream analysis so dashboards only show current data.
- Isolating stale sensor readings in an IoT ingestion job for anomaly review before archiving.
Sponsored
More from Data pipelines & processing
- Add a UUID Surrogate Key to Each Row in a CSV with Python easy
- Attach Source File Metadata to Records in Python easy
- Build a Python Utility That Detects Duplicate Records Across Multiple Excel Sheets medium
- Check Null Rate Threshold in PySpark DataFrame medium
- Count Records Processed per Category in Python easy
- Create Data Helper Functions in Python for Beginners easy
Keep learning
Related tutorials and quizzes for this topic.