How to Use as_completed to Process Futures in Order of Completion
Submit multiple tasks to a ThreadPoolExecutor and process each result as soon as it finishes using as_completed.
Python code
23 linesfrom concurrent.futures import ThreadPoolExecutor, as_completed
import time
def fetch_data(item_id):
time.sleep(1)
return f"item-{item_id}"
def main():
with ThreadPoolExecutor(max_workers=3) as executor:
future_map = {executor.submit(fetch_data, i): i for i in range(1, 6)}
for future in as_completed(future_map):
item_id = future_map[future]
try:
result = future.result()
print(f"Completed item {item_id}: {result}")
except Exception as exc:
print(f"Item {item_id} failed: {exc}")
if __name__ == "__main__":
main()Output
Completed item 2: item-2
Completed item 1: item-1
Completed item 4: item-4
Completed item 3: item-3
Completed item 5: item-5How it works
The as_completed iterator yields futures in the order they finish, not the order they were submitted, so results are ready as soon as each task completes. Calling future.result() on each yielded future gets the return value or re-raises any exception, which you catch with a try/except around the call. Creating a map from each future to its item_id lets you identify which task produced each result. The context manager (with) automatically shuts down the pool when the block exits, waiting for all tasks to finish.
Common mistakes
- Calling `future.result()` unconditionally without wrapping in try/except when tasks can raise
- Iterating over the future_map directly instead of using as_completed, which blocks on the first item
- Forgetting that as_completed yields futures non-deterministically and relying on submission order
Variations
- Use ProcessPoolExecutor for CPU-bound tasks to bypass the GIL
- Add a `timeout` argument to as_completed to cap waiting time on slow tasks
Real-world use cases
- Fetching multiple API endpoints concurrently and rendering each response as it arrives.
- Downloading files in parallel and processing the first one that completes to minimize latency.
- Running independent data-processing jobs and collecting results as jobs finish for live status updates.
Sponsored
More from Concurrency & performance
- Benchmark list.append vs deque.append in Python medium
- Build a Python Performance Profiler That Generates Readable Reports medium
- Graceful Shutdown Executor Context Manager in Python medium
- How to Build a Producer-Consumer Pattern with asyncio.Queue in Python medium
- How to Cancel an asyncio Task with Graceful Cleanup in Python medium
- How to Convert Data in Parallel with ThreadPoolExecutor in Python easy
Keep learning
Related tutorials and quizzes for this topic.