Redis

Redis as a Message Queue for Asynchronous Processing

June 9, 2026
11 min read

Sending an email, resizing an image, processing a payment — these tasks shouldn't block the HTTP response. Redis turns your existing cache infrastructure into a capable message queue with no extra dependencies, making async processing accessible to any application.

Redis Lists as a Simple Queue

Redis Lists are doubly-linked lists that support O(1) push and pop at both ends. This makes them a natural FIFO queue: producers push to one end, consumers pop from the other.

# Producer: push jobs onto the queue
LPUSH jobs:email '{"to":"user@example.com","subject":"Welcome"}'
LPUSH jobs:email '{"to":"other@example.com","subject":"Reset"}'

# Consumer: pop and process
RPOP jobs:email   # returns and removes the oldest item

LPUSH adds to the left (head), RPOP removes from the right (tail) — giving you first-in, first-out ordering.

Blocking Consumers with BRPOP

Polling with RPOP wastes CPU when the queue is empty. BRPOP blocks the connection until a message arrives, then returns it immediately.

# Python worker that blocks waiting for jobs
import redis
import json

r = redis.Redis(host='localhost', port=6379, decode_responses=True)

while True:
    # Block for up to 30 seconds, then loop
    result = r.brpop('jobs:email', timeout=30)
    if result:
        queue_name, payload = result
        job = json.loads(payload)
        send_email(job['to'], job['subject'])
        print(f"Sent email to {job['to']}")

BRPOP can watch multiple queues simultaneously — useful for priority queues where you check a high-priority queue before a low-priority one.

Reliable Queue with LMOVE

A plain BRPOP worker loses the job if it crashes mid-processing. The reliable queue pattern moves jobs to a processing list atomically before handling them.

# Atomically move from queue to processing list
LMOVE jobs:email jobs:email:processing RIGHT LEFT

# After successful processing, remove from processing list
LREM jobs:email:processing 1 '{"to":"user@example.com",...}'

# On startup, re-queue any stuck jobs
# (items still in jobs:email:processing from a crashed worker)
LRANGE jobs:email:processing 0 -1  # inspect stuck jobs

Redis Streams: A More Powerful Queue

Redis Streams (added in Redis 5.0) provide a persistent, append-only log with consumer groups — similar to Kafka but built into Redis.

# Producer: add message to stream
XADD jobs:stream * type email to user@example.com subject Welcome

# Create consumer group
XGROUP CREATE jobs:stream workers $ MKSTREAM

# Consumer: read new messages
XREADGROUP GROUP workers consumer1 COUNT 10 STREAMS jobs:stream >

# Acknowledge after processing
XACK jobs:stream workers 1686300000000-0

Streams give you:

  • Persistence — messages survive consumer restarts (unlike BRPOP)
  • Consumer groups — multiple workers share the load, each message delivered to one consumer
  • Acknowledgment — XACK confirms processing; unacknowledged messages can be reclaimed
  • History — replay messages from any point in the stream

When to Use Lists vs Streams

  • Redis Lists (BRPOP) — simple, low-volume queues where occasional message loss is acceptable. Perfect for background jobs, email queues, notification dispatch.
  • Redis Streams — higher reliability requirements, multiple consumer workers, need for message history or replay.

Production Elixir: off_broadway_redis_stream

If you are running Elixir, there is a community-driven Broadway producer specifically built for this use case: off_broadway_redis_stream. It wraps Redis Streams into a native Broadway pipeline, automatically managing consumer groups, distributed processing, backpressure, and fault tolerance — so you write business logic instead of plumbing.

Adding the Dependency

defp deps do
  [
    {:off_broadway_redis_stream, "~> 0.10.0"}
  ]
end

Implementation Template

Configure a Broadway pipeline to pull tasks concurrently out of a Redis Stream:

defmodule MyApp.OrderBroadway do
  use Broadway

  alias Broadway.Message

  def start_link(_opts) do
    Broadway.start_link(__MODULE__,
      name: __MODULE__,
      producer: [
        module: {OffBroadwayRedisStream.Producer, [
          redis_client_opts: [host: "localhost", port: 6379],
          stream: "orders:stream",
          group: "order_processing_group",
          consumer_name: "broadway_node_#{:rand.uniform(1000)}"
        ]}
      ],
      processors: [
        default: [
          concurrency: 3,
          min_demand: 1,
          max_demand: 10
        ]
      ]
    )
  end

  @impl true
  def handle_message(_processor, %Message{data: [_id, fields]} = message, _context) do
    payload = Map.new(Enum.chunk_every(fields, 2), fn [k, v] -> {k, v} end)

    IO.inspect(payload, label: "Broadway Processing Order")

    # Broadway automatically XACKs this message if no error is raised
    message
  end
end

The concurrency option maps directly to concurrent Elixir processes. Broadway's demand-driven model means each processor only pulls as many messages as it can handle, giving you built-in backpressure without any extra configuration.

How it Handles Failures

The real value of running Redis Streams inside Broadway comes from how off_broadway_redis_stream maps to the Redis primitives you have already seen:

  • Automatic XACK — if handle_message/3 finishes without raising, the adapter calls XACK automatically. You never write acknowledgment logic by hand.
  • Built-in XCLAIM failover — the library uses a heartbeat strategy to detect dead nodes. If a cluster node dies mid-processing, the remaining Broadway nodes detect the failure and call XCLAIM to take ownership of the pending messages, preventing them from being stuck in the PEL indefinitely.
  • Poison pill handling — if a message raises inside handle_message/3, Broadway's native handle_failed/2 callback intercepts it. Push the message to a dead-letter stream there and return it with Message.ack_immediately/1 to prevent the same message from crashing your pipeline in an infinite loop.

Key Takeaways

  • LPUSH + BRPOP is the simplest queue — producer pushes, worker blocks and pops
  • LMOVE provides at-least-once delivery by keeping jobs in a processing list until acknowledged
  • Redis Streams with XREADGROUP scales to multiple parallel workers with reliable delivery
  • off_broadway_redis_stream wraps Redis Streams in a Broadway pipeline for Elixir, handling XACK, XCLAIM failover, and backpressure automatically
  • Redis queues add no new infrastructure if you already use Redis for caching
  • For very high throughput or complex routing, dedicated brokers (RabbitMQ, Kafka) are better choices