How to Share a Queue Between Processes in Python

Use multiprocessing.Queue to pass work from a producer process to multiple consumer processes, coordinating with a sentinel stop message.

Medium Python 3.9+ Aug 9, 2026 Concurrency & performance 14 views 0 copies

multiprocessing queue concurrency inter-process

Python code

36 lines

Python 3.9+

import multiprocessing
import time


def producer(queue, items):
    for item in items:
        queue.put(item)
        time.sleep(0.1)
    queue.put("STOP")


def consumer(queue, name):
    while True:
        item = queue.get()
        if item == "STOP":
            break
        print(f"{name} processed: {item}")


if __name__ == "__main__":
    queue = multiprocessing.Queue()
    items = ["task-1", "task-2", "task-3"]

    p1 = multiprocessing.Process(target=producer, args=(queue, items))
    c1 = multiprocessing.Process(target=consumer, args=(queue, "consumer-1"))
    c2 = multiprocessing.Process(target=consumer, args=(queue, "consumer-2"))

    p1.start()
    c1.start()
    c2.start()

    p1.join()
    queue.put("STOP")
    queue.put("STOP")
    c1.join()
    c2.join()

Output

stdout

consumer-1 processed: task-1
consumer-2 processed: task-2
consumer-1 processed: task-3

How it works

multiprocessing.Queue provides a thread- and process-safe FIFO queue that lets multiple processes exchange Python objects without manual locking. The producer puts an item and then a sentinel "STOP"; however, with multiple consumers, a single sentinel is consumed by only one process, so the main process adds two extra STOP values before joining to ensure both consumers exit. Each consumer loops on queue.get() which blocks until data is available, printing each item. Joining the producer first guarantees all real items are queued before the shutdown signals are appended. This pattern scales to any number of consumers by matching the number of sentinels to the worker count.

Common mistakes

Sending only one sentinel for multiple consumers, causing some consumers to hang forever
Forgetting to call `join()` on producer before sending sentinels, leading to early exit
Using `queue.Empty` with blocking `get()` without timeout, which catches incorrectly
Creating the Queue inside a child process instead of passing it from the parent

Variations

Use `queue.get(timeout=1)` and catch `queue.Empty` to allow timeout-based shutdown
Use `multiprocessing.Pool` with `imap` for simpler producer-consumer patterns without manual sentinel management

Real-world use cases

Parallel web scraping where a producer fetches URLs and multiple worker processes parse HTML and store results.
Processing a large batch of images: one process reads file paths, others resize and save, scaling across CPU cores.
Building a logging pipeline where a producer gathers log lines and several consumers write to different sinks.

How to Share a Queue Between Processes in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Concurrency & performance

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Concurrency & performance

Keep learning

Tutorials

Quizzes