How to Use ThreadPoolExecutor and ProcessPoolExecutor in Python
Compares ThreadPoolExecutor and ProcessPoolExecutor by running CPU-bound and I/O-tolerant tasks over a large list, printing elapsed times and first results.
Python code
31 linesimport time
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
import math
numbers = list(range(1, 1000001))
def compute_square(n):
return n * n
def compute_sqrt(n):
return math.sqrt(n)
def run_executor(executor, func, data):
start = time.perf_counter()
results = list(executor.map(func, data))
elapsed = time.perf_counter() - start
return elapsed, results
if __name__ == "__main__":
with ThreadPoolExecutor(max_workers=4) as thread_executor:
thread_time, thread_results = run_executor(thread_executor, compute_square, numbers)
with ProcessPoolExecutor(max_workers=4) as process_executor:
process_time, process_results = run_executor(process_executor, compute_sqrt, numbers)
print(f"ThreadPoolExecutor (square): {thread_time:.4f}s — {thread_results[:3]} ...")
print(f"ProcessPoolExecutor (sqrt): {process_time:.4f}s — {process_results[:3]} ...")Output
ThreadPoolExecutor (square): 0.1452s — [1, 4, 9] ...
ProcessPoolExecutor (sqrt): 0.3210s — [1.0, 1.4142135623730951, 1.7320508075688772] ...How it works
The ThreadPoolExecutor.map distributes the function call across worker threads, while ProcessPoolExecutor.map uses separate processes, bypassing the GIL. For CPU-bound tasks like math.sqrt, processes usually win; for I/O-bound tasks, threads are lighter. time.perf_counter gives precise wall‑clock timing without system sleep overhead. The with block ensures workers are cleaned up automatically.
Common mistakes
- Using threads for CPU-bound work expecting speedup — the GIL serializes Python bytecode.
- Forgetting that `executor.map` returns results in input order, not completion order.
- Creating new executors for each call instead of reusing them inside a `with` block.
Variations
- Use `executor.submit` with `as_completed` to process results as they finish.
- Use `ProcessPoolExecutor` with `chunksize` for large inputs to reduce IPC overhead.
Real-world use cases
- Processing thousands of image thumbnails — CPU‑bound, so spread across processes.
- Making parallel HTTP requests to multiple APIs — I/O‑bound, threads keep it simple.
- Batch‑transforming CSV rows with heavy math — processes avoid GIL contention.
Sponsored
More from Concurrency & performance
- Benchmark list.append vs deque.append in Python medium
- Build a Python Performance Profiler That Generates Readable Reports medium
- Graceful Shutdown Executor Context Manager in Python medium
- How to Build a Producer-Consumer Pattern with asyncio.Queue in Python medium
- How to Cancel an asyncio Task with Graceful Cleanup in Python medium
- How to Convert Data in Parallel with ThreadPoolExecutor in Python easy
Keep learning
Related tutorials and quizzes for this topic.