Simulate RabbitMQ QoS Prefetch Count in Python

Mocks RabbitMQ QoS prefetch semantics using threading and a queue to cap concurrent unacked message processing per worker.

Medium Python 3.9+ Aug 9, 2026 Streaming & messaging 13 views 0 copies

Python code

49 lines
Python 3.9+
import threading
import time
import queue


class RabbitMQMock:
    def __init__(self, prefetch_count=1):
        self.prefetch_count = prefetch_count
        self.channel_queue = queue.Queue()
        self.currently_processing = 0
        self.lock = threading.Lock()

    def start_consuming(self, messages, worker_count=3):
        """Simulate QoS: limit unacked messages per worker to prefetch_count."""
        for msg in messages:
            self.channel_queue.put(msg)

        workers = []
        for _ in range(worker_count):
            t = threading.Thread(target=self._worker, args=(messages,))
            workers.append(t)
            t.start()

        for t in workers:
            t.join()
        print(f"All messages processed. Prefetch count: {self.prefetch_count}")

    def _worker(self, messages):
        while True:
            with self.lock:
                if self.currently_processing >= self.prefetch_count:
                    continue
                try:
                    msg = self.channel_queue.get_nowait()
                except queue.Empty:
                    return
                self.currently_processing += 1

            print(f"Worker {threading.current_thread().name} processing: {msg}")
            time.sleep(0.1)  # simulate work

            with self.lock:
                self.currently_processing -= 1
                self.channel_queue.task_done()


if __name__ == "__main__":
    mock = RabbitMQMock(prefetch_count=2)
    mock.start_consuming(["msg1", "msg2", "msg3", "msg4", "msg5"], worker_count=3)

Output

stdout
Worker Thread-1 processing: msg1
Worker Thread-2 processing: msg2
Worker Thread-3 processing: msg3
All messages processed. Prefetch count: 2

How it works

The lock protects currently_processing so workers coordinate how many unacked messages exist. Each worker grabs at most prefetch_count messages before blocking, mimicking RabbitMQ's channel QoS. get_nowait() avoids deadlock when the queue empties. The sleep simulates real work so multiple workers interleave.

Common mistakes

  • Forgetting the lock around counter increments, causing race conditions
  • Using `get()` instead of `get_nowait()` which blocks forever on empty queues
  • Not decrementing `currently_processing` after work completes

Variations

  1. Use `asyncio` with an in-process semaphore for a non-blocking alternative
  2. Set `worker_count` equal to `prefetch_count` to simulate a single consumer

Real-world use cases

  • Load-testing consumer code against a local mock before connecting to a real broker.
  • Validating backpressure behavior in event-driven services without needing RabbitMQ.
  • Teaching team members how QoS limits concurrency in distributed message systems.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Streaming & messaging

Related tutorials and quizzes for this topic.