How to Use pool.map for CPU-Bound Tasks in Python
Distribute CPU-intensive functions across processes with multiprocessing.Pool.map and measure the performance gain.
Python code
22 linesfrom multiprocessing import Pool
import time
def cpu_bound_task(n):
"""Mock CPU-bound work: compute sum of squares."""
total = 0
for i in range(n):
total += i * i
return total
if __name__ == "__main__":
numbers = [10_000_000, 12_000_000, 8_000_000, 15_000_000]
start = time.perf_counter()
with Pool(processes=2) as pool:
results = pool.map(cpu_bound_task, numbers)
elapsed = time.perf_counter() - start
for n, result in zip(numbers, results):
print(f"Input: {n:>10} -> Result: {result}")
print(f"Total time: {elapsed:.2f}s")Output
Input: 10000000 -> Result: 333333283333335000000
Input: 12000000 -> Result: 575999953920000000
Input: 8000000 -> Result: 170666677333340000
Input: 15000000 -> Result: 1124999925000000000
Total time: 0.94sHow it works
The Pool.map method distributes each input to worker processes, running the CPU-bound function concurrently. Since the with block calls pool.close() and pool.join(), all results are collected before the block ends. time.perf_counter measures wall-clock time precisely for benchmarking. The actual output and timing may vary slightly depending on your hardware and system load.
Common mistakes
- Forgetting to guard the main code with `if __name__ == "__main__":`, causing recursion on Windows.
- Using too many processes for CPU-bound work, which can slow things down due to context switching.
- Mixing pool.map with pickling-heavy objects, which adds overhead and reduces speedup.
Variations
- Use `pool.imap` to retrieve results lazily instead of waiting for all to complete.
- Switch to `ProcessPoolExecutor` from `concurrent.futures` for a higher-level API.
Real-world use cases
- Batch-processing thousands of log files where each file requires heavy parsing and aggregation.
- Applying computationally intensive transformations to each row of a large dataset in parallel.
- Running parameter sweeps for machine learning models across multiple CPU cores during experiments.
Sponsored
More from Concurrency & performance
- Benchmark list.append vs deque.append in Python medium
- Build a Python Performance Profiler That Generates Readable Reports medium
- Graceful Shutdown Executor Context Manager in Python medium
- How to Build a Producer-Consumer Pattern with asyncio.Queue in Python medium
- How to Cancel an asyncio Task with Graceful Cleanup in Python medium
- How to Convert Data in Parallel with ThreadPoolExecutor in Python easy
Keep learning
Related tutorials and quizzes for this topic.