Skip to content
Tutorials Center
My learning

Topics

Lesson 102 of 111 ยท Projects

Project: A Python App with Redis

Build a Python web app and a background worker that share a Redis queue, run both from one image with Compose, scale workers and stop them gracefully.

Many applications split their work in two: a web process that answers requests quickly, and workers that do slow jobs in the background. The two talk through a queue. In this project you will build exactly that with Python and Redis, run it with Docker Compose, and then scale the workers from one to three with a single flag.

You will learn how one image can serve two roles (only the command differs), how Redis acts as the queue, what --scale really does, and how to make workers stop gracefully. Everything was built and run on Docker 29 with Compose v5.

TEXT
 client --POST /jobs--> +-------+  RPUSH queue   +-------+  BLPOP queue  +----------+
                        |  web  | -------------> | redis | <------------ | worker 1 |
 client <--GET /jobs/N- +-------+  HSET job:N    +-------+               | worker 2 |
                                                     |                    | worker 3 |
                                                 volume "redisdata"       +----------+

The project layout

TEXT
queue-app/
  compose.yaml
  Dockerfile
  requirements.txt
  web.py
  worker.py

Step 1: The web process

The web process accepts a number, creates a job record, and puts the job id on a Redis list called queue. It does not do the slow work itself, so the HTTP response returns immediately with status 202 Accepted.

Python
# web.py
import json
import os
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

import redis

r = redis.Redis.from_url(os.environ["REDIS_URL"], decode_responses=True)


class Handler(BaseHTTPRequestHandler):
    def send_json(self, status, payload):
        body = json.dumps(payload).encode()
        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def do_POST(self):
        if self.path != "/jobs":
            return self.send_json(404, {"error": "not found"})
        length = int(self.headers.get("Content-Length", 0))
        number = int(json.loads(self.rfile.read(length) or b"{}").get("n", 0))
        job_id = r.incr("job:next")
        r.hset(f"job:{job_id}", mapping={"n": number, "status": "queued"})
        r.rpush("queue", job_id)
        self.send_json(202, {"id": job_id, "status": "queued"})

    def do_GET(self):
        if self.path == "/stats":
            self.send_json(200, {"queued": r.llen("queue"), "done": int(r.get("stats:done") or 0)})
        elif self.path.startswith("/jobs/"):
            job = r.hgetall(f"job:{self.path.split('/')[-1]}")
            self.send_json(200 if job else 404, job or {"error": "not found"})
        else:
            self.send_json(404, {"error": "not found"})

    def log_message(self, fmt, *args):
        print("web:", fmt % args, flush=True)


if __name__ == "__main__":
    print("web: listening on 8000", flush=True)
    ThreadingHTTPServer(("0.0.0.0", 8000), Handler).serve_forever()

Each job is a Redis hash (job:1, job:2, ...) and the queue holds only job ids. INCR hands out unique ids atomically, so several web processes would never clash.

Step 2: The worker

The worker loops forever: wait for an id on the queue, mark the job as running, "work" for one second, store the result. BLPOP is a blocking pop. It sleeps inside Redis until something arrives (we ask for a 2-second timeout so the loop can notice a shutdown request), so idle workers use almost no CPU, and when several workers wait on the same list Redis hands each job to exactly one of them.

Python
# worker.py
import os
import signal
import socket
import time

import redis

r = redis.Redis.from_url(os.environ["REDIS_URL"], decode_responses=True)
name = socket.gethostname()
running = True


def stop(signum, frame):
    global running
    running = False
    print(f"worker {name}: got signal {signum}, finishing the current job", flush=True)


signal.signal(signal.SIGTERM, stop)
signal.signal(signal.SIGINT, stop)


def is_prime(n):
    if n < 2:
        return False
    return all(n % d for d in range(2, int(n ** 0.5) + 1))


print(f"worker {name}: ready", flush=True)
while running:
    item = r.blpop("queue", timeout=2)
    if item is None:
        continue
    job_id = item[1]
    number = int(r.hget(f"job:{job_id}", "n"))
    r.hset(f"job:{job_id}", "status", "running")
    time.sleep(1)  # pretend this is slow work
    r.hset(f"job:{job_id}", mapping={"status": "done", "prime": str(is_prime(number)).lower(), "worker": name})
    r.incr("stats:done")
    print(f"worker {name}: job {job_id}: {number} prime={is_prime(number)}", flush=True)

print(f"worker {name}: bye", flush=True)

Three Docker-specific habits are hiding in this file:

  • flush=True on every print. Python buffers output when it is not a terminal, and docker logs would show nothing until the buffer fills. The Dockerfile below also sets PYTHONUNBUFFERED=1 for the same reason.
  • A SIGTERM handler. docker stop sends SIGTERM. A process running as PID 1 gets no default action for signals it has not registered a handler for, so it would only be killed after the grace period (Signals, PID 1 and --init). Our handler sets a flag so the loop ends after the current job.
  • The host name as an identity. Inside a container the host name is the short container id, which gives each worker a unique name to put into the log lines and results.

Step 3: One image, two commands

The web process and the worker use the same code and libraries, so they share one image. Only the command differs.

TEXT
# requirements.txt
redis==6.2.0
DOCKERFILE
FROM python:3.13-slim

ENV PYTHONUNBUFFERED=1 PIP_NO_CACHE_DIR=1
WORKDIR /app

COPY requirements.txt .
RUN pip install -r requirements.txt

COPY web.py worker.py ./

RUN useradd --system --uid 10001 app
USER app

CMD ["python", "web.py"]

The default CMD starts the web process; Compose overrides it for the worker (CMD and ENTRYPOINT). Dependencies come before code so changing worker.py does not reinstall Redis's client (The Build Cache), and the app runs as an unprivileged user (Users and Permissions).

Step 4: The Compose file

YAML
# compose.yaml
name: tcd-b15-queue

services:
  redis:
    image: redis:alpine
    command: ["redis-server", "--appendonly", "yes"]
    volumes:
      - redisdata:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5

  web:
    build: .
    image: tcd-b15-queue-app
    ports:
      - "127.0.0.1:18152:8000"
    environment:
      REDIS_URL: redis://redis:6379/0
    depends_on:
      redis:
        condition: service_healthy

  worker:
    build: .
    image: tcd-b15-queue-app
    command: ["python", "worker.py"]
    environment:
      REDIS_URL: redis://redis:6379/0
    depends_on:
      redis:
        condition: service_healthy
    restart: unless-stopped

volumes:
  redisdata:

What to notice:

  • image: tcd-b15-queue-app on both services gives the one build a single name. Without it Compose would tag two identical images ...-web and ...-worker.
  • Redis persistence is on (--appendonly yes plus the redisdata volume mounted at /data). A queue holds work that must not vanish, unlike the cache in Project: A Full-Stack App with Compose.
  • The worker has no ports and no container_name. Both would stop it from being scaled: two containers cannot share a published port or a fixed name.
  • Only web is published, on loopback. Redis is reachable inside the network under the name redis and nowhere else.

Step 5: Start it

CONSOLE
$ docker compose config --quiet && echo ok
ok
$ docker compose up -d --build
 Image tcd-b15-queue-app Built
 Network tcd-b15-queue_default Created
 Volume tcd-b15-queue_redisdata Created
 Container tcd-b15-queue-redis-1 Started
 Container tcd-b15-queue-redis-1 Healthy
 Container tcd-b15-queue-worker-1 Started
 Container tcd-b15-queue-web-1 Started
$ docker compose ps
NAME                     IMAGE               COMMAND                  SERVICE   CREATED          STATUS                    PORTS
tcd-b15-queue-redis-1    redis:alpine        "docker-entrypoint.sโ€ฆ"   redis     19 seconds ago   Up 12 seconds (healthy)   6379/tcp
tcd-b15-queue-web-1      tcd-b15-queue-app   "python web.py"          web       13 seconds ago   Up 5 seconds              127.0.0.1:18152->8000/tcp
tcd-b15-queue-worker-1   tcd-b15-queue-app   "python worker.py"       worker    13 seconds ago   Up 6 seconds

Both web and worker run the same image (tcd-b15-queue-app) with different commands, and only web has a published port.

Step 6: Send jobs to one worker

Submit six numbers and ask for statistics as the single worker chews through them (one second per job):

CONSOLE
$ for n in 97 100 101 7919 8000 104729; do curl -s -X POST -d "{\"n\":$n}" http://127.0.0.1:18152/jobs; echo; done
{"id": 1, "status": "queued"}
{"id": 2, "status": "queued"}
{"id": 3, "status": "queued"}
{"id": 4, "status": "queued"}
{"id": 5, "status": "queued"}
{"id": 6, "status": "queued"}
$ curl -s http://127.0.0.1:18152/stats
{"queued": 5, "done": 0}
$ sleep 3; curl -s http://127.0.0.1:18152/stats
{"queued": 2, "done": 3}
$ sleep 4; curl -s http://127.0.0.1:18152/stats
{"queued": 0, "done": 6}
$ curl -s http://127.0.0.1:18152/jobs/1
{"n": "97", "status": "done", "prime": "true", "worker": "aa3ebae1b2af"}

The POSTs came back instantly while the work took about six seconds: that is the point of a queue. The worker's log shows the jobs in order:

CONSOLE
$ docker compose logs --no-log-prefix worker
worker aa3ebae1b2af: ready
worker aa3ebae1b2af: job 1: 97 prime=True
worker aa3ebae1b2af: job 2: 100 prime=False
worker aa3ebae1b2af: job 3: 101 prime=True
worker aa3ebae1b2af: job 4: 7919 prime=True
worker aa3ebae1b2af: job 5: 8000 prime=False
worker aa3ebae1b2af: job 6: 104729 prime=True

Step 7: Scale the workers

One worker is the bottleneck. --scale starts more containers of the same service; nothing else changes:

CONSOLE
$ docker compose up -d --scale worker=3
 Container tcd-b15-queue-worker-1 Running
 Container tcd-b15-queue-worker-3 Started
 Container tcd-b15-queue-worker-2 Started
$ docker compose ps worker --format 'table {{.Name}}\t{{.Status}}'
NAME                     STATUS
tcd-b15-queue-worker-1   Up 47 seconds
tcd-b15-queue-worker-2   Up 11 seconds
tcd-b15-queue-worker-3   Up 13 seconds

Submit ten more jobs. With three workers they take about four seconds instead of ten, and the log (an excerpt is shown) spreads the jobs across all three container names:

CONSOLE
$ docker compose logs --no-log-prefix worker | grep -E "job (1[0-6]|[7-9]):"
worker 362144ba22c6: job 7: 11 prime=True
worker 362144ba22c6: job 10: 14 prime=False
worker 362144ba22c6: job 13: 17 prime=True
worker aa3ebae1b2af: job 8: 12 prime=False
worker aa3ebae1b2af: job 11: 15 prime=False
worker aa3ebae1b2af: job 14: 18 prime=False
worker 8b5a9372474d: job 9: 13 prime=True
worker 8b5a9372474d: job 12: 16 prime=False

(The lines come from separate containers, so their order in the combined output can differ from run to run.) No load balancer was needed: the workers simply compete for items on the same Redis list, which is why queues scale so naturally. Compare this with Containers Talking to Each Other, where a client needs to find a single server.

Without --no-log-prefix, docker compose logs tags each line with the container (worker-1 | ...), which helps when you follow all workers at once with docker compose logs -f worker.

Step 8: Stop gracefully, lose nothing

Stopping the workers shows why we handled SIGTERM:

CONSOLE
$ time docker compose stop worker
 Container tcd-b15-queue-worker-1 Stopped
 Container tcd-b15-queue-worker-2 Stopped
 Container tcd-b15-queue-worker-3 Stopped

real	0m1.7s
$ docker compose logs worker | grep -E "signal|bye"
worker-1  | worker aa3ebae1b2af: got signal 15, finishing the current job
worker-1  | worker aa3ebae1b2af: bye
worker-3  | worker 362144ba22c6: got signal 15, finishing the current job
worker-3  | worker 362144ba22c6: bye
worker-2  | worker 8b5a9372474d: got signal 15, finishing the current job
worker-2  | worker 8b5a9372474d: bye

All three workers saw signal 15, finished the job in hand and said goodbye in under two seconds. Without the handler each would have been killed after the default 10-second timeout, possibly in the middle of a job.

What happens to jobs sent while no worker is running? They wait in Redis:

CONSOLE
$ docker compose stop worker
$ for n in 31 32 33; do curl -s -o /dev/null -X POST -d "{\"n\":$n}" http://127.0.0.1:18152/jobs; done
$ curl -s http://127.0.0.1:18152/stats
{"queued": 3, "done": 22}
$ docker compose start worker
$ sleep 4; curl -s http://127.0.0.1:18152/stats
{"queued": 0, "done": 25}

The web tier kept accepting work while the workers were down, and the backlog drained as soon as one came back. That decoupling is the main reason for using a queue.

This queue is intentionally simple, and it has a real weakness. BLPOP removes the id from the list before the worker starts. If a worker is killed with SIGKILL, loses power, or crashes mid-job, that job is gone from the queue and stays in the state running forever. Production systems use reliable queues (a Redis list plus a "processing" list, Redis Streams with acknowledgements, or a library such as Celery, RQ or Dramatiq) that re-deliver unfinished jobs.

Step 9: Persistence and the scale reset

Remove the whole stack without -v, bring it back and look at the data:

CONSOLE
$ docker compose down
$ docker compose up -d
$ curl -s http://127.0.0.1:18152/stats
{"queued": 0, "done": 22}
$ curl -s http://127.0.0.1:18152/jobs/22
{"n": "26", "status": "done", "prime": "false", "worker": "362144ba22c6"}

The counters and finished jobs survived because Redis wrote them to the append-only file on the redisdata volume (Volumes). Another detail: the stack came back with one worker. A plain docker compose up -d returns every service to the count in the file (one by default), so keep scale in the file when you want it to stick:

YAML
  worker:
    deploy:
      replicas: 3

Finally, confirm the processes are not root:

CONSOLE
$ docker compose exec worker id
uid=10001(app) gid=999(app) groups=999(app)

Clean up

CONSOLE
$ docker compose down -v --rmi local
 Network tcd-b15-queue_default Removed
 Image tcd-b15-queue-app:latest Removed

Where to take it next

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why must the worker service have no ports mapping and no container_name before you scale it?

    Two containers cannot bind the same host port or share one fixed name. Without them, Compose creates worker-1, worker-2, ... automatically.
  2. 2. How do three workers avoid processing the same job twice?

    Redis list operations are atomic. Whichever worker's BLPOP is served first receives the id; the others keep waiting for the next one.
  3. 3. The web container and the worker container both use the image tcd-b15-queue-app. How do they behave differently?

    One image can serve several roles. CMD in the Dockerfile starts the web process, and the command: key of the worker service replaces it.
  4. 4. What is the purpose of the SIGTERM handler in worker.py?

    docker stop sends SIGTERM and waits (10 seconds by default) before sending SIGKILL. The handler sets a flag so the loop ends after the current job.
  5. 5. A worker is killed with SIGKILL in the middle of a job. What happens to that job in this project?

    A simple BLPOP queue gives at-most-once delivery. Reliable queues keep a record of in-flight work so it can be re-delivered.

Summary

  • A web process plus background workers talking through a queue is a very common pattern, and Compose runs both from one image with different command: values.
  • Redis lists make a simple queue: RPUSH to add, BLPOP to wait for and take work. Several workers share the load without any load balancer.
  • docker compose up -d --scale worker=3 runs more copies; services to be scaled must not publish a fixed port or set container_name. Put deploy.replicas: 3 under the service to make the count permanent (checked: up -d then starts three workers).
  • Print with flush=True (or set PYTHONUNBUFFERED) so logs appear immediately, and handle SIGTERM so containers stop in a second instead of ten.
  • Turn on Redis persistence and a volume when the queue holds work you cannot lose, and remember that a simple queue still loses jobs when a worker is killed.

Next: Project: Multiple Apps Behind a Proxy