How to Implement Backpressure Pause Producer with a Bounded Queue in Python
Places a Producer thread that sends items into a bounded queue with backpressure: on Full, it pauses to let the consumer catch up.
Python code
44 linesimport threading
import time
import queue
import random
class Producer:
def __init__(self, q):
self.q = q
self.running = True
def produce(self):
while self.running:
item = random.randint(1, 100)
try:
self.q.put(item, timeout=0.5)
print(f"Produced: {item}, qsize={self.q.qsize()}", flush=True)
except queue.Full:
print(f"Backpressure: queue full (size={self.q.qsize()}), pausing...", flush=True)
time.sleep(0.2) # simulated backpressure pause
def stop(self):
self.running = False
def consumer(q, num_items):
for _ in range(num_items):
item = q.get()
time.sleep(0.1) # simulate processing
q.task_done()
print(f"Consumed: {item}", flush=True)
if __name__ == "__main__":
q = queue.Queue(maxsize=3)
p = Producer(q)
producer_thread = threading.Thread(target=p.produce)
producer_thread.start()
consumer_thread = threading.Thread(target=consumer, args=(q, 8))
consumer_thread.start()
consumer_thread.join()
p.stop()
producer_thread.join()
Output
Produced: 42, qsize=1
Produced: 17, qsize=2
Produced: 93, qsize=3
Backpressure: queue full (size=3), pausing...
Consumed: 42
Consumed: 17
Produced: 56, qsize=2
Consumed: 93
...
How it works
The Queue with maxsize=3 enforces a bound: put blocks when full, but timeout=0.5 turns that block into a Full exception. Catching Full allows the producer to log a backpressure event and sleep(0.2) to slow down, simulating a pause. The consumer runs in another thread, pulling items with get and calling task_done after processing. This interleaving shows a classic bounded-buffer pattern with explicit backpressure handling.
Common mistakes
- Forgetting `task_done()` — the `join()` never returns and threads hang.
- Setting a large `maxsize` thinking it prevents deadlock, but ignoring backpressure events.
- Not using `flush=True` in prints, causing output to appear out of order or lost in threads.
- Calling `stop()` while producer is blocked in `put` without timeout — it never checks the flag.
Variations
- Use `q.put_nowait(item)` and catch `queue.Full` without a timeout block.
- Implement a semaphore-based rate limiter to pause the producer instead of `sleep`.
Real-world use cases
- In an ETL pipeline, a producer reads from a database and writes to a queue that feeds a slow consumer — pausing avoids memory overload.
- In a web scraper, a producer dispatches URLs to a bounded queue and pauses when the processing workers lag behind.
- In a message ingestion service, a producer places events into a bounded buffer to backpressure when the downstream sink is slow.
Sponsored
More from Streaming & messaging
- At Most Once Fire-and-Forget Mock in Python easy
- Batch Consume Process Commit Pattern in Python medium
- Build a Streaming Messaging Helper in Python easy
- Dead Letter Queue Failed Messages List Mock in Python easy
- Dedupe processed message IDs in Python easy
- Event Envelope with Schema Version Field in Python easy
Keep learning
Related tutorials and quizzes for this topic.