Skip to content
Tutorials Center
My learning

Topics

Lesson 45 of 48 ยท Projects

Project: Job Queue with Supervision

Build a background job queue in Elixir: a queue process with back-pressure, workers that retry failed jobs and a supervisor that restarts crashes.

Almost every real application needs background work: sending emails, resizing images, calling slow APIs. A job queue accepts work quickly and lets workers do it later. It is also a perfect project for learning OTP, because a good queue must survive failing jobs, crashing workers and floods of new work.

In this project you will design a queue with three safety features: retries for jobs that fail, supervisor restarts for workers that crash and back-pressure that refuses new work when the queue is full. Elixir is not installed on this site, so the real code is read-only. Each idea also has a JavaScript stand-in you can run. You should know GenServer, Supervisors and Supervision Trees and Tasks and Parallel Work.

Three safety nets

Retries A failed job gets another try.
Supervisor A crashed worker is restarted.
Back-pressure A full queue says no.

Each net catches a different kind of failure.

1 / 2

Design

The queue is a GenServer whose state is a list or :queue of pending jobs. Clients call enqueue/1; workers ask for the next job. A supervisor starts the queue and a fixed number of workers.

flowchart LR
    C["Clients"] -->|"enqueue"| Q["Queue<br/>(GenServer)"]
    Q -->|"next job"| W1["Worker 1"]
    Q -->|"next job"| W2["Worker 2"]
    S["Supervisor"] -.->|"restarts"| W1
    S -.->|"restarts"| W2
    S -.->|"restarts"| Q

The supervisor sits above everything and restarts whatever fails.

Two decisions shape the whole design:

  1. Workers pull, the queue does not push. A worker asks for a job only when it is free. That way a slow worker never receives a pile of messages in its mailbox.
  2. The queue has a maximum size. enqueue/1 returns {:error, :full} when there is no room, and the caller decides what to do (retry later, drop, show an error). This is back-pressure: pressure from a full system travels back to the producer instead of silently using more memory.

The queue

ELIXIR
defmodule JobQueue do
  use GenServer

  @max_size 100

  def start_link(opts \\ []) do
    GenServer.start_link(__MODULE__, :queue.new(), Keyword.put_new(opts, :name, __MODULE__))
  end

  def enqueue(job), do: GenServer.call(__MODULE__, {:enqueue, job})
  def next, do: GenServer.call(__MODULE__, :next)

  @impl true
  def init(queue), do: {:ok, queue}

  @impl true
  def handle_call({:enqueue, job}, _from, queue) do
    if :queue.len(queue) >= @max_size do
      {:reply, {:error, :full}, queue}
    else
      {:reply, :ok, :queue.in(job, queue)}
    end
  end

  def handle_call(:next, _from, queue) do
    case :queue.out(queue) do
      {{:value, job}, rest} -> {:reply, {:ok, job}, rest}
      {:empty, queue} -> {:reply, :empty, queue}
    end
  end
end

:queue is an Erlang module for a first-in, first-out queue with cheap operations at both ends (a plain list is slow to append to; see Tuples and Lists). :queue.in/2 adds at the back and :queue.out/1 takes from the front. Using call for enqueue gives the client an answer, either :ok or the back-pressure error.

Retries

A job is a map with a function to run and a counter for attempts. The worker catches failures and decides whether to try again:

ELIXIR
defmodule JobWorker do
  @max_attempts 3

  def run(%{fun: fun, attempts: attempts} = job) do
    try do
      {:ok, fun.()}
    rescue
      error ->
        if attempts + 1 >= @max_attempts do
          {:dead, error}
        else
          JobQueue.enqueue(%{job | attempts: attempts + 1})
          :retry
        end
    end
  end
end

The pattern %{fun: fun, attempts: attempts} = job pulls out two fields and also keeps the whole map. %{job | attempts: attempts + 1} is the update syntax and returns a new map with one key changed. After the third failed attempt the job is declared dead, and a real system would store it somewhere (a "dead letter" list) for a human to inspect. Without a limit, a job that can never succeed would loop forever. Production systems also wait before retrying, usually with exponential backoff, using Process.send_after/3.

Here is the same logic with an async JavaScript worker:

JS Retries and dead jobs (JavaScript stand-in)

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Job 1 succeeds at once, job 2 succeeds on its third attempt, and job 3 never succeeds and is declared dead after three attempts. Retried jobs go to the back of the queue, so other jobs are not blocked behind a failing one.

Back-pressure

Change the sample above to enqueue ten jobs and you will see { error: "full" } for those that do not fit. The caller must handle that result:

ELIXIR
case JobQueue.enqueue(job) do
  :ok -> :accepted
  {:error, :full} -> {:error, :try_again_later}
end

In a web application this becomes an HTTP 503 or 429 response with a Retry-After header. A queue that accepts everything is not safe: when producers outrun workers, memory grows until the node dies. Task.async_stream/3 has built-in back-pressure through its max_concurrency option, and GenStage and Broadway generalise the pull-based idea.

Supervision

Retries handle jobs that raise errors. They do not help when a worker process itself dies, for instance from an exit signal or a bug outside the try. That is the supervisor's job:

ELIXIR
defmodule JobSystem do
  use Supervisor

  def start_link(_), do: Supervisor.start_link(__MODULE__, :ok, name: __MODULE__)

  @impl true
  def init(:ok) do
    children = [
      JobQueue,
      Supervisor.child_spec({JobConsumer, 1}, id: :consumer_1),
      Supervisor.child_spec({JobConsumer, 2}, id: :consumer_2)
    ]

    Supervisor.init(children, strategy: :rest_for_one, max_restarts: 3, max_seconds: 5)
  end
end

The strategy :rest_for_one matters. Children start in list order, and if one crashes, the supervisor restarts it and every child started after it. The queue is first, so if the queue dies, the consumers are restarted too (they would otherwise hold a reference to a dead queue). If a single consumer crashes, only that consumer restarts.

max_restarts: 3, max_seconds: 5 is a safety valve. If more than three restarts happen within five seconds, the supervisor decides that restarting does not help, shuts down its children and crashes itself, escalating the problem to its own supervisor. Restarts cure transient problems; they cannot fix a bug that fails every time.

A restarted queue starts empty, so pending jobs are lost. For jobs that must never be lost, persist them (a database table is the usual choice) and let the queue reload them in init/1. Libraries such as Oban do exactly that.

In the stand-in below, supervise restarts a failing worker and gives up after too many restarts, like max_restarts:

JS A supervisor with a restart limit (JavaScript stand-in)

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The flaky worker recovers after two restarts. The broken worker is restarted three times and then the supervisor stops trying and reports shutdown. In real OTP the supervisor would also exit with the reason :shutdown, and a parent supervisor would apply its own strategy.

Using the system

ELIXIR
{:ok, _} = JobSystem.start_link([])

JobQueue.enqueue(%{fun: fn -> IO.puts("sending email") end, attempts: 0})

In a Mix application you would put JobSystem in the children list of your Application module, so it starts with the app (see Mix: Creating and Managing Projects). Test the pure parts (the retry decision) directly and use ExUnit's start_supervised!/1 for the processes (see Testing with ExUnit).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is back-pressure in this queue?

    Back-pressure pushes the 'I am full' signal back to producers instead of letting memory grow without bound.
  2. 2. Why do workers pull jobs instead of the queue pushing them?

    With pull, a worker takes a job only when it is free, so the work is spread by readiness.
  3. 3. With :rest_for_one, what happens when the first child crashes?

    The supervisor restarts the failed child and every child that comes after it in the list, because later children may depend on earlier ones.
  4. 4. What does max_restarts: 3, max_seconds: 5 protect against?

    If the limit is exceeded, the supervisor gives up and exits so the failure escalates instead of looping forever.
  5. 5. What is lost when the queue process is restarted by its supervisor?

    A restarted process begins again from init/1; durable jobs must be stored outside the process, for example in a database.

Summary

  • A job queue combines three failure strategies: retries for failing jobs, supervisors for dying processes and back-pressure for overload.
  • Workers pull jobs, and the queue has a size limit and answers {:error, :full} when it is reached.
  • Limit retries, then move the job to a dead-letter store; add backoff in real systems.
  • Choose a restart strategy that matches dependencies (:rest_for_one when later children need earlier ones) and set max_restarts.
  • Jobs that must not be lost need storage outside the process.

Next up: Practice Exercises with Solutions.