How to Use ThreadPoolExecutor in Python for Parallel Processing
Use ThreadPoolExecutor with executor.map to run a function over many inputs concurrently and collect ordered results.
Python code
10 linesdef worker(item):
return item * item
if __name__ == "__main__":
from concurrent.futures import ThreadPoolExecutor
numbers = list(range(1, 11))
with ThreadPoolExecutor(max_workers=4) as executor:
results = list(executor.map(worker, numbers))
print("Input: ", numbers)
print("Results:", results)Output
Input: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
Results: [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]How it works
The concurrent.futures.ThreadPoolExecutor manages a pool of worker threads. Executor.map submits each item from the iterable to the pool and returns results in the same order as the input, preserving determinism. Using the context manager ensures threads are joined and resources cleaned up even if an exception occurs. The function 'worker' squares each number, but this pattern works for any CPU-bound or I/O-bound helper.
Common mistakes
- Using ThreadPoolExecutor for CPU-bound tasks; GIL limits speedup, consider ProcessPoolExecutor.
- Forgetting to wrap in a context manager, leaving threads unmanaged.
- Assuming results are returned in input order; map ensures order but async can scramble.
- Creating a new executor for every call; reuse one for multiple batches.
Variations
- Use as_completed to process results as they finish, not in input order.
- Switch to ProcessPoolExecutor for true parallelism on CPU-heavy work.
Real-world use cases
- Fetches multiple URLs concurrently in a web scraper, collecting responses in posts order.
- Processes a batch of image thumbnails in parallel using threads for I/O-like file operations.
- Runs several database queries simultaneously and aggregates the results in one flow.
Sponsored
More from Concurrency & performance
- Benchmark list.append vs deque.append in Python medium
- Build a Python Performance Profiler That Generates Readable Reports medium
- Graceful Shutdown Executor Context Manager in Python medium
- How to Build a Producer-Consumer Pattern with asyncio.Queue in Python medium
- How to Cancel an asyncio Task with Graceful Cleanup in Python medium
- How to Convert Data in Parallel with ThreadPoolExecutor in Python easy
Keep learning
Related tutorials and quizzes for this topic.