How to Use multiprocessing Pool map and starmap in Python
Parallelize functions over iterables with Pool.map, and unpack multiple arguments via Pool.starmap.
Python code
20 linesfrom multiprocessing import Pool
def square(x):
return x * x
def add_and_multiply(a, b, c):
return (a + b) * c
if __name__ == "__main__":
numbers = [1, 2, 3, 4, 5]
with Pool(processes=2) as pool:
squares = pool.map(square, numbers)
print(f"squares: {squares}")
starmap_args = [(1, 2, 3), (4, 5, 6), (7, 8, 9)]
results = pool.starmap(add_and_multiply, starmap_args)
print(f"starmap results: {results}")Output
squares: [1, 4, 9, 16, 25]
starmap results: [9, 54, 135]How it works
Pool.map applies a single-argument function to each item of the iterable, distributing the work across the worker processes. Pool.starmap works like map but unpacks each argument tuple into the function's parameters, allowing multi-argument functions. Using with Pool(...) as pool: ensures the pool is properly closed and joined after the block, preventing leaked processes. The if __name__ == "__main__": guard is required on Windows and recommended everywhere to avoid recursive process spawning. Both methods return results in the same order as the input, blocking until all tasks complete.
Common mistakes
- Forgetting the `if __name__ == '__main__':` guard on Windows causing infinite recursion.
- Using `pool.apply` or `pool.map` for multi-argument functions instead of `starmap`.
- Calling `pool.map` with a generator, which still loads all items into memory.
- Not using a context manager, risking orphaned pool processes.
Variations
- Use `pool.map_async` and `pool.starmap_async` for non-blocking calls and retrieving results later.
- Use `Pool(processes=cpu_count())` with `os.cpu_count()` to utilize all cores.
Real-world use cases
- Processing each row of a large CSV file with a CPU-intensive transformation in parallel.
- Computing similarity scores for many pairs of vectors where each pair requires multiple parameters.
- Batching image thumbnail generation across a folder of images using a multi-argument resize function.
Sponsored
More from Concurrency & performance
- Benchmark list.append vs deque.append in Python medium
- Build a Python Performance Profiler That Generates Readable Reports medium
- Graceful Shutdown Executor Context Manager in Python medium
- How to Build a Producer-Consumer Pattern with asyncio.Queue in Python medium
- How to Cancel an asyncio Task with Graceful Cleanup in Python medium
- How to Convert Data in Parallel with ThreadPoolExecutor in Python easy
Keep learning
Related tutorials and quizzes for this topic.