How to spawn multiple worker processes in Python with multiprocessing.Process
Spawns three separate worker processes using multiprocessing.Process, runs them concurrently, and waits for all to finish before printing a completion message.
Python code
20 linesimport multiprocessing
import time
def worker(name):
print(f"Worker {name} started")
time.sleep(1)
print(f"Worker {name} finished")
return name
if __name__ == "__main__":
processes = []
for i in range(3):
p = multiprocessing.Process(target=worker, args=(i,))
processes.append(p)
p.start()
for p in processes:
p.join()
print("All workers completed")Output
Worker 0 started
Worker 1 started
Worker 2 started
Worker 0 finished
Worker 1 finished
Worker 2 finished
All workers completedHow it works
This code creates three multiprocessing.Process objects, each targeting the worker function with a different argument. Calling start() on each process launches it in a separate OS process, allowing the CPU to run them concurrently. The join() method blocks until each process has finished, ensuring the main program waits for all workers. The if __name__ == "__main__" guard is crucial on Windows and some Unix platforms to avoid recursive process creation.
Common mistakes
- Forgetting the `if __name__ == '__main__'` guard, causing errors on Windows.
- Calling `join()` inside the creation loop, which serializes the processes.
- Assuming print output is completely ordered (it can interleave).
Variations
- Use `multiprocessing.Pool` and `pool.map()` for simpler parallel mapping.
- Pass extra arguments using keyword arguments with `Process(target=worker, kwargs={'name': i})`.
Real-world use cases
- Parallelizing CPU-bound tasks like image processing across multiple cores.
- Running multiple independent data validation checks concurrently in a data pipeline.
- Spawning separate worker processes for each file in a batch ETL job.
Sponsored
More from Concurrency & performance
- Benchmark list.append vs deque.append in Python medium
- Build a Python Performance Profiler That Generates Readable Reports medium
- Graceful Shutdown Executor Context Manager in Python medium
- How to Build a Producer-Consumer Pattern with asyncio.Queue in Python medium
- How to Cancel an asyncio Task with Graceful Cleanup in Python medium
- How to Convert Data in Parallel with ThreadPoolExecutor in Python easy
Keep learning
Related tutorials and quizzes for this topic.