How to Use Thread Pool Executor map for IO-Bound Tasks in Python
Run multiple I/O-bound tasks concurrently with ThreadPoolExecutor map and collect their results in order.
Python code
16 linesimport time
from concurrent.futures import ThreadPoolExecutor
def io_bound_task(task_id: int) -> str:
time.sleep(0.2) # mock I/O wait
return f"Task {task_id} completed"
def main() -> None:
task_ids = [1, 2, 3, 4, 5]
with ThreadPoolExecutor(max_workers=3) as executor:
results = list(executor.map(io_bound_task, task_ids))
for result in results:
print(result)
if __name__ == "__main__":
main()Output
Task 1 completed
Task 2 completed
Task 3 completed
Task 4 completed
Task 5 completedHow it works
ThreadPoolExecutor creates a pool of worker threads that run tasks concurrently. The map method applies a function to each item in the iterable, distributing work across threads while preserving result order. The with block ensures all threads are joined and resources cleaned up after execution. For I/O-bound tasks (network calls, disk reads, API requests), threads help overlap the waiting time — the limiting factor is time.sleep, not CPU, so thread switching gives a parallel speedup. Results from map come back in the same order as the input iterable, even if tasks finish out of order.
Common mistakes
- Using threads for CPU-heavy work where the GIL prevents real parallelism
- Forgetting to wrap `executor.map` in `list()` — map returns a lazy iterator
- Creating a new executor per task instead of reusing one pool for many jobs
Variations
- Use `executor.submit` with `as_completed` to process results as they finish rather than waiting for all
- Use `ProcessPoolExecutor` instead when the task is CPU-bound and needs true parallelism
Real-world use cases
- Batching HTTP requests to an external API and collecting all responses before processing them
- Reading multiple files or database queries in parallel during a data pipeline stage
- Calling several third-party services for enrichment and aggregating their results for one request
Sponsored
More from Concurrency & performance
- Benchmark list.append vs deque.append in Python medium
- Build a Python Performance Profiler That Generates Readable Reports medium
- Graceful Shutdown Executor Context Manager in Python medium
- How to Build a Producer-Consumer Pattern with asyncio.Queue in Python medium
- How to Cancel an asyncio Task with Graceful Cleanup in Python medium
- How to Convert Data in Parallel with ThreadPoolExecutor in Python easy
Keep learning
Related tutorials and quizzes for this topic.