Use Redis as a message queue

You've built a fast API, your services communicate synchronously, and everything works — until a traffic spike hits. Suddenly, your background jobs timeout, email confirmations never send, and users wait forever for reports. The root cause? Coupling request handling to time-consuming work. Using Redis as a message queue decouples those tasks, letting you fire and forget work items that get processed asynchronously. In this lesson, you'll master the LPUSH/BRPOP pattern to turn Redis into a lightweight, durable message queue — no extra infrastructure required.

The problem this lesson solves

When your application handles tasks like sending emails, resizing images, or generating PDFs synchronously, the user's request blocks until the work finishes. Under load, this causes:

• Timeouts – HTTP requests exceed acceptable limits.
• Poor user experience – pages hang for seconds.
• Resource exhaustion – too many concurrent workers fight for memory and CPU.

Dedicated message brokers like RabbitMQ or Apache Kafka are powerful, but overkill for many projects. Redis, which you may already run for caching, can serve as a message queue with no additional setup. The trade-off? You trade some durability guarantees for simplicity and speed.

By the end of this section, you'll understand why "using Redis as a message queue" is a pragmatic choice for many backend systems, especially in prototyping or moderate-scale production.

Core concept / mental model

Think of Redis as a bucket with a line. Producers drop tasks (messages) into one end of the line; consumers pick tasks from the other end. This first-in, first-out (FIFO) behavior is exactly what a basic queue provides.

Key definitions

• Producer: Any process that adds a message to the queue.
• Consumer: A process that pulls and processes messages.
• Queue: A Redis list key. The left/right ends become the queue ends.
• Blocking pop: A consumer that waits for a message to appear, rather than polling.

Why Redis lists work as queues

Redis lists are ordered by insertion order. Two commands form the backbone:

• LPUSH – add to the left (head).
• RPOP – remove from the right (tail).

Together, they implement a FIFO queue. For consumers that need to wait efficiently, BRPOP blocks until a message is available, avoiding busy loops.

Pro tip: The pattern is LPUSH + BRPOP. LPUSH adds work; BRPOP waits and retrieves. This is the simplest, most common way to use Redis as a message queue.

How it works step by step

Let's trace a complete job lifecycle.

Step 1: Producer adds a message

A web request comes in — e.g., a user signs up. The producer serializes the task (e.g., send_welcome_email) and LPUSHes it to a list key like queue:email.

import redis

r = redis.Redis(host='localhost', port=6379, db=0)
task = {"email": "user@example.com", "type": "welcome"}
r.lpush('queue:email', str(task))

Step 2: Consumer waits for messages

A background worker runs an infinite loop, calling BRPOP. It blocks until a message arrives, then acknowledges and processes it.

while True:
    # Blocks until a message is available (timeout=0 means forever)
    queue, message = r.brpop('queue:email', timeout=0)
    process_email_task(message)  # your custom logic

Step 3: Processing and acknowledgment

Once BRPOP returns, the message is removed from Redis. The consumer must ensure it processes the message before crashing. If the consumer crashes after BRPOP but before processing, the message is lost. This is a core reliability challenge we'll address later.

Step 4: Scaling consumers

Multiple consumers can read from the same queue. Each message goes to exactly one consumer because BRPOP removes it atomically. This gives you natural horizontal scaling.

# Run two worker processes on the same machine
python worker.py &
python worker.py &

Hands-on walkthrough

Now you'll set up a real queue with Python and Redis. Make sure Redis is running locally. Install the redis-py library:

pip install redis

Example 1: Basic producer-consumer

Producer (producer.py)

import redis
import json

r = redis.Redis(decode_responses=True)

# Simulate 3 tasks
for i in range(3):
    task = {"task_id": i, "payload": f"data-{i}"}
    r.lpush('myqueue', json.dumps(task))
    print(f"Produced: {task}")

Consumer (consumer.py)

import redis
import json

r = redis.Redis(decode_responses=True)

while True:
    # Block until message arrives
    msg = r.brpop('myqueue', timeout=0)
    if msg:
        # msg is (key, value) tuple
        task = json.loads(msg[1])
        print(f"Consumed: {task}")
        # Simulate processing
        # NOTE: if you crash here, the task is lost!

Run producer once, then start consumer:

python producer.py
python consumer.py

Expected output (consumer):

Consumed: {'task_id': 0, 'payload': 'data-0'}
Consumed: {'task_id': 1, 'payload': 'data-1'}
Consumed: {'task_id': 2, 'payload': 'data-2'}

Example 2: Adding basic reliability with BRPOPLPUSH

Redis 6.2+ supports BRPOPLPUSH or the newer BLMOVE. These atomically move the message from the main queue to a backup list. If processing fails, you can recover the backup.

import redis
import json

r = redis.Redis(decode_responses=True)

while True:
    # Atomically move from queue:backlog to queue:processing
    msg = r.brpoplpush('queue:backlog', 'queue:processing', timeout=0)
    if msg:
        task = json.loads(msg)
        try:
            process_task(task)
            # On success, remove from processing list
            r.lrem('queue:processing', 1, msg)
        except Exception as e:
            print(f"Failed: {e}")
            # Message remains in queue:processing for manual inspection

This pattern gives you at-least-once delivery in many cases, though duplicates are still possible.

Compare options / when to choose what

Not all queue solutions are equal. Here's how Redis compares to dedicated brokers:

When to choose Redis:

• You already run Redis for caching.
• You need low latency for small payloads.
• Loss of a few tasks is acceptable (e.g., cache invalidation).
• You're prototyping or have moderate throughput (<1000 msg/s).

When to avoid Redis:

• You need strict at-least-once or exactly-once delivery.
• Tasks must survive Redis restarts without loss (use AOF + fsync always).
• You need complex routing or delayed messages.

Troubleshooting & edge cases

Common pitfalls

1. Message loss on consumer crash - Issue: Consumer dies after BRPOP but before processing. - Fix: Use BRPOPLPUSH/BLMOVE to keep a backup queue. Implement periodic cleanup of stuck messages.

2. Queue buildup - Issue: Producers faster than consumers. Memory usage grows. - Fix: Add consumers, or use a rate limiter in the producer (e.g., Redis THROTTLE with RediSearch).

3. Duplicate processing - Issue: Same message processed twice (e.g., after failover). - Fix: Design tasks to be idempotent — store a processed set (e.g., SADD result after success).

4. Blocking BRPOP with multiple queues - Issue: Need to listen to several queues efficiently. - Fix: Use BRPOP with multiple keys — it returns the first available. Or use BLPOP.

# Listen to three queues; prioritize queue1
task = r.brpop(['queue1', 'queue2', 'queue3'], timeout=30)

1. Memory limits - Redis evicts keys when memory is full. Your queue key may get evicted! - Fix: Set maxmemory-policy noeviction for queue databases, or use separate Redis instance.

What you learned & what's next

You now understand how to use Redis as a message queue with the LPUSH/BRPOP pattern. You can:

• Explain the mental model: producers push, consumers block-pop.
• Write a producer-consumer system in Python.
• Add basic reliability using BRPOPLPUSH.
• Compare Redis queues to RabbitMQ and Kafka.
• Troubleshoot common issues like message loss and queue backpressure.

Next lesson: Redis Streams — a more advanced data structure that offers consumer groups, message IDs, and persistence. Streams are ideal when you need at-least-once delivery without external tools.

Go forth and decouple your services — your users (and your sleep schedule) will thank you.