How to Share a Dict and List Between Processes with multiprocessing Manager in Python
This code demonstrates how to share a dictionary and a list between multiple processes using multiprocessing.Manager, enabling safe concurrent updates.
Python code
29 linesimport multiprocessing as mp
def worker(shared_dict, shared_list, name):
shared_dict[name] = name.upper()
shared_list.append(name)
print(f"{name} added to shared structures")
def main():
with mp.Manager() as manager:
shared_dict = manager.dict()
shared_list = manager.list()
processes = []
for name in ["alpha", "beta", "gamma"]:
p = mp.Process(target=worker, args=(shared_dict, shared_list, name))
processes.append(p)
p.start()
for p in processes:
p.join()
print("Final shared dict:", dict(shared_dict))
print("Final shared list:", list(shared_list))
if __name__ == "__main__":
main()Output
alpha added to shared structures
beta added to shared structures
gamma added to shared structures
Final shared dict: {'alpha': 'ALPHA', 'beta': 'BETA', 'gamma': 'GAMMA'}
Final shared list: ['alpha', 'beta', 'gamma']How it works
multiprocessing.Manager creates server processes that host shared objects, allowing multiple processes to read and write them safely. The dict() and list() methods return proxy objects that handle synchronization internally. Each worker function updates the shared structures and prints confirmation, so the final state reflects all updates. Using with mp.Manager() ensures the manager server is properly shut down after all processes complete.
Common mistakes
- Forgetting to join processes before reading the final shared state, causing incomplete data.
- Assuming native dict/list operations are atomic with Manager proxies; they are synchronized but not always atomic for compound operations.
- Passing the Manager object itself instead of the individual proxy objects to worker processes.
Variations
- Use `manager.Namespace()` to create a shared object with attributes that can be updated.
- Use `multiprocessing.Array` and `multiprocessing.Value` for simpler shared primitives without Manager overhead.
Real-world use cases
- Aggregating results from parallel workers into a shared dictionary for a MapReduce-style job.
- Maintaining a shared configuration store that multiple processes update during a distributed simulation.
- Collecting logs or metrics from multiple child processes into a shared list for centralized processing.
Sponsored
More from Concurrency & performance
- Benchmark list.append vs deque.append in Python medium
- Build a Python Performance Profiler That Generates Readable Reports medium
- Graceful Shutdown Executor Context Manager in Python medium
- How to Build a Producer-Consumer Pattern with asyncio.Queue in Python medium
- How to Cancel an asyncio Task with Graceful Cleanup in Python medium
- How to Convert Data in Parallel with ThreadPoolExecutor in Python easy
Keep learning
Related tutorials and quizzes for this topic.