Implement an Out-of-Order Sort Buffer with a Heap in Python
Buffers out-of-order indices from a stream and emits them in sorted order using a min-heap with a sliding window.
Python code
38 linesimport heapq
from collections import deque
class OutOfOrderSorter:
def __init__(self, buffer_size):
self.buffer_size = buffer_size
self.buffer = deque(maxlen=buffer_size)
self.heap = []
self.next_expected_index = 0
self.result = []
def push(self, item):
heapq.heappush(self.heap, item)
self._flush_ready()
def _flush_ready(self):
while self.heap and self.heap[0] == self.next_expected_index:
item = heapq.heappop(self.heap)
self.result.append(item)
self.next_expected_index += 1
def finish(self):
while self.heap:
item = heapq.heappop(self.heap)
self.result.append(item)
return self.result
if __name__ == "__main__":
sorter = OutOfOrderSorter(buffer_size=3)
# items arrive out of order: (index, value)
stream = [(3, "c"), (1, "a"), (2, "b"), (0, "start"), (5, "e"), (4, "d")]
for idx, val in stream:
sorter.push(idx)
print(f"Buffered indices: {sorter.buffer}")
print(f"Heap: {sorter.heap}")
print(f"Sorted output: {sorter.finish()}")
Output
Buffered indices: deque([5, 4], maxlen=3)
Heap: []
Sorted output: [0, 1, 2, 3, 4, 5]
How it works
The heapq module maintains a min-heap so the smallest index is always at the top. _flush_ready pops items as soon as the smallest heap element matches the expected sequential index, producing sorted output without waiting for the entire stream. The deque(maxlen=buffer_size) keeps only the most recent buffer window for diagnostics. Using a heap avoids sorting the whole buffer each time, giving O(log n) per push. finish drains any remaining heap items to complete the sequence.
Common mistakes
- Forgetting to flush when new items arrive, causing the heap to grow unboundedly.
- Calling finish() before all items are pushed, truncating the output.
- Assuming the buffer deque is used for sorting—it is only for inspection.
Variations
- Replace the deque buffer with a simple list of recent indices for smaller memory footprint.
- Use `heapq.merge` for multiple sorted streams to produce global order.
Real-world use cases
- Reordering out-of-order messages from a distributed queue before further processing.
- Assembling video frames arriving in the wrong sequence for a streaming pipeline.
- Merging time-series data points that arrive with skew from multiple sensors.
Sponsored
More from Data pipelines & processing
- Add a UUID Surrogate Key to Each Row in a CSV with Python easy
- Attach Source File Metadata to Records in Python easy
- Build a Python Utility That Detects Duplicate Records Across Multiple Excel Sheets medium
- Check Null Rate Threshold in PySpark DataFrame medium
- Count Records Processed per Category in Python easy
- Create Data Helper Functions in Python for Beginners easy
Keep learning
Related tutorials and quizzes for this topic.