Simulate RabbitMQ QoS Prefetch Count in Python
Mocks RabbitMQ QoS prefetch semantics using threading and a queue to cap concurrent unacked message processing per worker.
Python code
49 linesimport threading
import time
import queue
class RabbitMQMock:
def __init__(self, prefetch_count=1):
self.prefetch_count = prefetch_count
self.channel_queue = queue.Queue()
self.currently_processing = 0
self.lock = threading.Lock()
def start_consuming(self, messages, worker_count=3):
"""Simulate QoS: limit unacked messages per worker to prefetch_count."""
for msg in messages:
self.channel_queue.put(msg)
workers = []
for _ in range(worker_count):
t = threading.Thread(target=self._worker, args=(messages,))
workers.append(t)
t.start()
for t in workers:
t.join()
print(f"All messages processed. Prefetch count: {self.prefetch_count}")
def _worker(self, messages):
while True:
with self.lock:
if self.currently_processing >= self.prefetch_count:
continue
try:
msg = self.channel_queue.get_nowait()
except queue.Empty:
return
self.currently_processing += 1
print(f"Worker {threading.current_thread().name} processing: {msg}")
time.sleep(0.1) # simulate work
with self.lock:
self.currently_processing -= 1
self.channel_queue.task_done()
if __name__ == "__main__":
mock = RabbitMQMock(prefetch_count=2)
mock.start_consuming(["msg1", "msg2", "msg3", "msg4", "msg5"], worker_count=3)
Output
Worker Thread-1 processing: msg1
Worker Thread-2 processing: msg2
Worker Thread-3 processing: msg3
All messages processed. Prefetch count: 2
How it works
The lock protects currently_processing so workers coordinate how many unacked messages exist. Each worker grabs at most prefetch_count messages before blocking, mimicking RabbitMQ's channel QoS. get_nowait() avoids deadlock when the queue empties. The sleep simulates real work so multiple workers interleave.
Common mistakes
- Forgetting the lock around counter increments, causing race conditions
- Using `get()` instead of `get_nowait()` which blocks forever on empty queues
- Not decrementing `currently_processing` after work completes
Variations
- Use `asyncio` with an in-process semaphore for a non-blocking alternative
- Set `worker_count` equal to `prefetch_count` to simulate a single consumer
Real-world use cases
- Load-testing consumer code against a local mock before connecting to a real broker.
- Validating backpressure behavior in event-driven services without needing RabbitMQ.
- Teaching team members how QoS limits concurrency in distributed message systems.
Sponsored
More from Streaming & messaging
- At Most Once Fire-and-Forget Mock in Python easy
- Batch Consume Process Commit Pattern in Python medium
- Build a Streaming Messaging Helper in Python easy
- Dead Letter Queue Failed Messages List Mock in Python easy
- Dedupe processed message IDs in Python easy
- Event Envelope with Schema Version Field in Python easy
Keep learning
Related tutorials and quizzes for this topic.