How to Use ProcessPoolExecutor for CPU Parallel Map in Python
Run a function over a sequence of inputs in parallel across multiple CPU cores with ProcessPoolExecutor.map.
Python code
25 linesfrom concurrent.futures import ProcessPoolExecutor
import math
def compute_square(num):
return num * num
def is_prime(n):
if n < 2:
return False
for i in range(2, int(math.sqrt(n)) + 1):
if n % i == 0:
return False
return True
if __name__ == "__main__":
numbers = range(1, 11)
with ProcessPoolExecutor() as executor:
squares = list(executor.map(compute_square, numbers))
with ProcessPoolExecutor() as executor:
primes = list(executor.map(is_prime, range(2, 21)))
print("Squares:", squares)
print("Prime check:", primes)Output
Squares: [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
Prime check: [False, True, True, False, True, False, True, False, False, False, True, False, True, False, False, False, True, False, True, False]How it works
ProcessPoolExecutor.map distributes each input item to a worker process, running the callable in parallel across CPU cores. The if __name__ == "__main__" guard is mandatory so worker processes on Windows and macOS can safely import the script. Each with block waits for all tasks to finish and collects results in input order before exiting. Use this for CPU-bound tasks like arithmetic or prime checks, where the GIL would otherwise serialize threads. Results come back as a list when you wrap the map in list(), preserving the original order of the inputs.
Common mistakes
- Forgetting the `if __name__ == "__main__"` guard, which causes infinite recursion or crashes on Windows/macOS.
- Using ProcessPoolExecutor for I/O-bound tasks where asyncio or ThreadPoolExecutor would be much faster.
- Calling `executor.map` but never consuming the iterator, so tasks never start.
- Sharing large picklable objects between workers via closures instead of passing them through arguments, slowing marshalling.
Variations
- Use `executor.submit` with `as_completed` when you need to handle results as they finish instead of in order.
- Raise the worker count with `ProcessPoolExecutor(max_workers=os.cpu_count() - 1)` to reserve a core for the main process.
Real-world use cases
- Batch-resizing or filtering thousands of images in a web service without blocking the event loop.
- Running cryptographic hash rounds or checksum verification across log files in a data pipeline.
- Parallelizing computational geometry or simulation sweeps on a reporting server with idle cores.
Sponsored
More from Concurrency & performance
- Benchmark list.append vs deque.append in Python medium
- Build a Python Performance Profiler That Generates Readable Reports medium
- Graceful Shutdown Executor Context Manager in Python medium
- How to Build a Producer-Consumer Pattern with asyncio.Queue in Python medium
- How to Cancel an asyncio Task with Graceful Cleanup in Python medium
- How to Convert Data in Parallel with ThreadPoolExecutor in Python easy
Keep learning
Related tutorials and quizzes for this topic.