Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 36 ยท Brokers and Backends

Using Redis as a Celery Broker: URLs and Visibility Timeout

Configure Redis as a Celery broker: URL formats, databases, Sentinel, TLS, the visibility timeout that causes duplicate tasks, memory and persistence concerns.

Redis is the broker most people start with. It is simple to run, fast and often already present in the stack. It is also a data store that was not designed as a message broker, so a few behaviours need attention: how URLs are formed, how long an unacknowledged message waits before being redelivered, and what happens when Redis restarts.

Broker URLs

Python
app.conf.broker_url = "redis://localhost:6379/0"
app.conf.broker_url = "redis://:[email protected]:6379/0"        # password only
app.conf.broker_url = "redis://user:[email protected]:6379/0"    # ACL user
app.conf.broker_url = "rediss://redis.example.com:6380/0"            # TLS
app.conf.broker_url = "redis+socket:///var/run/redis/redis.sock"     # Unix socket

Format: redis://[user]:[password]@host:port/db. The path part is the database number (0 to 15 on a default Redis). With rediss:// (two s) the connection uses TLS; also set certificate requirements:

Python
import ssl
app.conf.broker_use_ssl = {"ssl_cert_reqs": ssl.CERT_REQUIRED}
app.conf.redis_backend_use_ssl = {"ssl_cert_reqs": ssl.CERT_REQUIRED}

Passwords with special characters (@, /, :, #) must be percent-encoded in the URL. This runnable check parses a URL the way a client would, and shows the encoding:

Python Parsing and building a Redis broker URL

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

How Celery uses Redis keys

Understanding the keys helps when you debug with redis-cli.

  • Each queue is a Redis list. The default queue is the list named celery. LLEN celery is the number of waiting messages, the single most useful metric.
  • Priority levels use extra lists named like celery\x06\x163.
  • Messages a worker has received but not yet acknowledged live in a hash named unacked, with an index unacked_index (a sorted set by time).
  • Worker control messages use Redis Pub/Sub channels.
  • Delayed (ETA/countdown) tasks are held in the worker's memory until their time comes; they are unacknowledged in Redis in the meantime.
Terminal
redis-cli -n 0 LLEN celery            # waiting tasks
redis-cli -n 0 HLEN unacked           # delivered but not yet acked
redis-cli -n 0 LRANGE celery 0 0      # peek at the oldest message (JSON)

The visibility timeout

Redis has no built-in "this message is leased to consumer X" feature. Celery fakes it: when a worker receives a message, it is moved into unacked. If the worker does not acknowledge it within the visibility timeout (one hour by default), Celery assumes the worker died and puts the message back in the queue, where another worker runs it.

That has two sharp consequences:

  1. Long ETA/countdown tasks are redelivered. A task with countdown=7200 is unacknowledged for two hours in the worker's memory. After one hour Redis puts it back, and it will run twice. The same happens to tasks that run longer than the timeout under acks_late=True.
  2. If you raise it too far, crash recovery gets slow. A dead worker's messages return only after the timeout.

Set it larger than your longest ETA or run time:

Python
app.conf.broker_transport_options = {"visibility_timeout": 43200}   # 12 hours, in seconds

The simulation below runs a short task with a short timeout so you can see redelivery. Time is simulated.

Python Stand-in: the visibility timeout

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In the second scenario the task is still running when the lease expires. Worker B receives a second copy, and the report is built twice. That is exactly what an under-sized visibility timeout does in production.

High availability

A single Redis is a single point of failure. Options:

  • Redis Sentinel: automatic failover with one primary and replicas.

    Python
    app.conf.broker_url = "sentinel://s1:26379;sentinel://s2:26379;sentinel://s3:26379"
    app.conf.broker_transport_options = {"master_name": "mymaster"}
    
  • Managed Redis (ElastiCache, Memorystore, and similar), which handles failover for you.

  • Redis Cluster is not supported as a Celery broker; use Sentinel or a managed primary-replica setup.

Persistence, memory and eviction

Redis keeps data in memory. If it restarts without persistence, queued tasks vanish. Enable AOF (appendonly yes, appendfsync everysec) when losing a few seconds of tasks is acceptable and a restart must not wipe the queue. Also:

  • Set maxmemory-policy noeviction on the broker instance. An eviction policy such as allkeys-lru would silently delete queued tasks when memory is full. With noeviction, Redis returns errors instead, which is visible and safer.
  • Do not share one Redis between the broker and a cache that uses eviction. Use separate instances or databases with separate policies; the eviction policy is per instance.
  • Watch memory: millions of queued messages need gigabytes.

Tuning worker connections

Python
app.conf.broker_transport_options = {
    "visibility_timeout": 43200,
    "socket_timeout": 5,
    "socket_connect_timeout": 5,
    "health_check_interval": 30,
}
app.conf.broker_connection_retry_on_startup = True
app.conf.broker_pool_limit = 10        # connections kept by the producer

Web workers (gunicorn with many threads) that all call delay() can open many Redis connections. Cap broker_pool_limit and maxclients accordingly.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the default Redis list name for Celery's default queue?

    The default queue is called celery, so LLEN celery shows the waiting tasks.
  2. 2. A task with countdown=7200 runs twice with Redis defaults. Why?

    Raise visibility_timeout above the longest countdown or run time (or use RabbitMQ for long delays).
  3. 3. Which eviction policy is appropriate for a Redis instance used as a broker?

    Eviction would silently delete queued messages. noeviction makes Redis reject writes when full, which you can detect.
  4. 4. How do you write a password containing @ in a broker URL?

    Reserved URL characters must be percent-encoded, otherwise the URL parses incorrectly.

Summary

  • Use redis://[user]:[password]@host:port/db; rediss:// for TLS; percent-encode special characters.
  • Queues are lists (LLEN celery); unacknowledged messages sit in the unacked hash.
  • The visibility timeout (default 1 hour) redelivers messages that are unacknowledged too long, so raise it above your longest ETA or run time.
  • Use Sentinel or a managed service for failover, noeviction for the broker instance, and AOF persistence when restarts must not drop tasks.

Next: Using RabbitMQ: Exchanges, Queues and Routing Keys