Balance load with prefetch and fair dispatch

Your workers are idle while others grind under a mountain of messages, and you can't figure out why. The culprit is often a naive fetch strategy that grabs one message at a time or, conversely, one that lets a slow worker hoard a huge batch. In this lesson, you'll master balance load with prefetch and fair dispatch — two of the most powerful knobs in messaging systems like RabbitMQ — so you can squeeze maximum throughput from your worker pool without sacrificing responsiveness.

The problem this lesson solves

When you connect multiple consumers to the same queue, the broker's default behavior can wreck your latency and throughput. Here's what goes wrong:

• Uneven work distribution: Some workers end up with a backlog while others sit idle. This happens because the broker hands out messages one at a time by default, but a slow consumer can still get flooded if it prefetches a large batch.
• Head-of-line blocking: A worker that fetches 50 messages in advance can block behind one slow task, leaving the other 49 waiting even though other workers are free.
• Unfair scheduling: By default, RabbitMQ dispatches messages in a round-robin fashion without considering how fast each consumer processes them. This leads to the classic fair dispatch problem: a fast worker gets one message, finishes quickly, and then has to wait for the next round while a slow worker holds up the queue.

If you've ever seen a CPU spike on one instance while another sits at 5%, you've felt this pain. The fix is to configure prefetch count and enable fair dispatch — two settings that let the broker act as a load balancer for your workers.

Core concept / mental model

Think of your message broker as a dispatcher at a busy call center. Without any rules, the dispatcher hands a call to the first available agent, then moves to the next agent in line, regardless of how long each call takes. Fast agents finish quickly and twiddle their thumbs while slow agents talk for hours. That's unfair dispatch.

Fair dispatch flips the script: the broker only sends a new message to a worker after it has acknowledged the previous one. The broker tracks how many unacknowledged messages each worker has, and it won't send more than the worker's prefetch limit.

Prefetch count is the maximum number of unacknowledged messages a consumer can have at any time. Set it to 1 and you get true fair dispatch — each worker handles one message at a time. Set it higher (say 10 or 50) and you allow a worker to buffer messages locally, which boosts throughput but risks uneven load if tasks vary in duration.

The key insight: prefetch and fair dispatch are two sides of the same coin. Fair dispatch is achieved by setting a low prefetch count, and prefetch tuning is how you balance throughput against fairness.

How it works step by step

Here's the flow in a RabbitMQ-like broker when you configure fair dispatch:

1. Consumer connects and sets a prefetch count — usually via a channel method like basic_qos(prefetch_count=1).
2. Broker tracks unacknowledged messages per consumer — it maintains a count of messages sent but not yet acked.
3. Consumer processes a message — does the work, then sends an acknowledgment (either auto-ack or manual ack).
4. Broker decrements the unacked count and now considers the consumer eligible for the next message.
5. Broker dispatches a new message only if the consumer's unacked count is below the prefetch limit.

In RabbitMQ, you also need basic_consume with auto_ack=False (manual acknowledgment) for fair dispatch to work. If you use auto-ack, the broker treats the message as acked immediately on delivery, so it never knows how long a consumer actually takes — fair dispatch is impossible.

Hands-on walkthrough

Let's put this into practice with a minimal RabbitMQ example using pika. First, make sure you have RabbitMQ running locally (e.g., via Docker: docker run -d --name rabbitmq -p 5672:5672 rabbitmq:3).

Step 1: Set up a sender

Create a sender.py that publishes 20 messages with a simulated processing cost indicator:

import pika

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
channel.queue_declare(queue='tasks', durable=True)

for i in range(20):
    message = f"task-{i}"
    channel.basic_publish(exchange='', routing_key='tasks', body=message.encode())
    print(f"Published {message}")

connection.close()

Step 2: Create a worker with fair dispatch

Now build worker.py that consumes with prefetch_count=1 and manual acks:

import pika, time

def callback(ch, method, properties, body):
    print(f"Received {body.decode()}")
    time.sleep(2)  # Simulate a slow task
    ch.basic_ack(delivery_tag=method.delivery_tag)

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
channel.queue_declare(queue='tasks', durable=True)

# Fair dispatch: only one unacked message per worker
channel.basic_qos(prefetch_count=1)
channel.basic_consume(queue='tasks', on_message_callback=callback, auto_ack=False)

print('Worker ready. Press CTRL+C to exit.')
channel.start_consuming()

Run the sender, then launch three workers in separate terminals. Watch how each worker receives exactly one message, then waits for ack before the next. The broker distributes tasks round-robin but only to workers that have finished their previous task — that's fair dispatch.

Step 3: Observe the difference

If you set prefetch_count=10 instead, one worker will grab up to 10 messages and process them sequentially while the others sit idle. Run the same script with different prefetch values and watch the console output — you'll see the imbalance immediately.

Compare options / when to choose what

Choose prefetch=1 when: - Tasks have widely varying durations. - You have many workers and want minimal idle time. - You need predictable latency.

Choose a higher prefetch when: - Tasks are uniform and fast. - Network round-trip time dominates (you want to amortize it). - You're processing batches and can tolerate delays.

Troubleshooting & edge cases

• Fair dispatch not working: Make sure you set auto_ack=False and call basic_ack in the callback. If you use auto-ack, the broker thinks the message is done instantly, so it will keep sending more — prefetch is ignored.
• Unbalanced load despite prefetch=1: Check that your consumers are actually running and connected to the same queue. Also verify that you set basic_qos on the correct channel before consuming.
• Message loss on crash: If a worker crashes after fetching but before acking, the message is requeued — but only if auto_ack=False. Without manual ack, it's lost forever.
• Prefetch too high causing memory spikes: A high prefetch buffers messages in memory. If your messages are large, monitor worker RAM. Lower prefetch if you see OOM errors.
• Dead-lettering and requeue: If a message fails repeatedly, it can be requeued infinitely. Pair fair dispatch with a max-retry count or a dead-letter queue to avoid infinite loops.

What you learned & what's next

You now know how to balance load with prefetch and fair dispatch — you can explain the problem, apply basic_qos(prefetch_count=1) in pika, and choose the right prefetch value for your workload. You've also seen how manual acks are essential for fairness.

Next in the Messaging & queues track, you'll move up to dead-letter choreography — how to handle messages that can't be processed, with retries, routing, and DLQ exchanges. This builds directly on what you've learned here about message lifecycle and acknowledgment. Get ready to make your queues resilient, not just fair.