Redis as a Message Queue for Asynchronous Processing

Sending an email, resizing an image, processing a payment — these tasks shouldn't block the HTTP response. Redis turns your existing cache infrastructure into a capable message queue with no extra dependencies, making async processing accessible to any application.

Redis Lists as a Simple Queue

Redis Lists are doubly-linked lists that support O(1) push and pop at both ends. This makes them a natural FIFO queue: producers push to one end, consumers pop from the other.

# Producer: push jobs onto the queue
LPUSH jobs:email '{"to":"user@example.com","subject":"Welcome"}'
LPUSH jobs:email '{"to":"other@example.com","subject":"Reset"}'

# Consumer: pop and process
RPOP jobs:email   # returns and removes the oldest item

LPUSH adds to the left (head), RPOP removes from the right (tail) — giving you first-in, first-out ordering.

Blocking Consumers with BRPOP

Polling with RPOP wastes CPU when the queue is empty. BRPOP blocks the connection until a message arrives, then returns it immediately.

# Python worker that blocks waiting for jobs
import redis
import json

r = redis.Redis(host='localhost', port=6379, decode_responses=True)

while True:
    # Block for up to 30 seconds, then loop
    result = r.brpop('jobs:email', timeout=30)
    if result:
        queue_name, payload = result
        job = json.loads(payload)
        send_email(job['to'], job['subject'])
        print(f"Sent email to {job['to']}")

BRPOP can watch multiple queues simultaneously — useful for priority queues where you check a high-priority queue before a low-priority one.

Reliable Queue with LMOVE

A plain BRPOP worker loses the job if it crashes mid-processing. The reliable queue pattern moves jobs to a processing list atomically before handling them.

# Atomically move from queue to processing list
LMOVE jobs:email jobs:email:processing RIGHT LEFT

# After successful processing, remove from processing list
LREM jobs:email:processing 1 '{"to":"user@example.com",...}'

# On startup, re-queue any stuck jobs
# (items still in jobs:email:processing from a crashed worker)
LRANGE jobs:email:processing 0 -1  # inspect stuck jobs

Redis Streams: A More Powerful Queue

Redis Streams (added in Redis 5.0) provide a persistent, append-only log with consumer groups — similar to Kafka but built into Redis.

# Producer: add message to stream
XADD jobs:stream * type email to user@example.com subject Welcome

# Create consumer group
XGROUP CREATE jobs:stream workers $ MKSTREAM

# Consumer: read new messages
XREADGROUP GROUP workers consumer1 COUNT 10 STREAMS jobs:stream >

# Acknowledge after processing
XACK jobs:stream workers 1686300000000-0

Streams give you:

Persistence — messages survive consumer restarts (unlike BRPOP)
Consumer groups — multiple workers share the load, each message delivered to one consumer
Acknowledgment — XACK confirms processing; unacknowledged messages can be reclaimed
History — replay messages from any point in the stream

When to Use Lists vs Streams

Redis Lists (BRPOP) — simple, low-volume queues where occasional message loss is acceptable. Perfect for background jobs, email queues, notification dispatch.
Redis Streams — higher reliability requirements, multiple consumer workers, need for message history or replay.

Key Takeaways

LPUSH + BRPOP is the simplest queue — producer pushes, worker blocks and pops
LMOVE provides at-least-once delivery by keeping jobs in a processing list until acknowledged
Redis Streams with XREADGROUP scales to multiple parallel workers with reliable delivery
Redis queues add no new infrastructure if you already use Redis for caching
For very high throughput or complex routing, dedicated brokers (RabbitMQ, Kafka) are better choices