Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 36 ยท Fault Tolerance and OTP

Supervisors and supervision trees

Learn OTP supervisors: child specs, one_for_one, one_for_all and rest_for_one strategies, restart intensity, temporary and transient children, and supervision trees.

A link tells a process that a partner died. A supervisor is a process whose only job is to watch other processes, its children, and restart them according to rules. It contains no business logic, so it is simple and very unlikely to crash. Combine supervisors into a supervision tree and you get programs that heal themselves.

A first supervisor

A supervisor is a module with the supervisor behaviour and a single callback, init/1, which returns the strategy and a list of child specifications.

ERLANG
-module(kv_sup).
-behaviour(supervisor).

-export([start_link/0, init/1]).

start_link() ->
    supervisor:start_link({local, ?MODULE}, ?MODULE, []).

init([]) ->
    SupFlags = #{strategy => one_for_one,
                 intensity => 3,
                 period => 10},
    Children = [
        #{id => kv,
          start => {kv, start_link, []},
          restart => permanent,
          shutdown => 5000,
          type => worker,
          modules => [kv]}
    ],
    {ok, {SupFlags, Children}}.

supervisor:start_link/3 starts the supervisor, which calls init/1, then starts every child by calling its start function ({Module, Function, Args}, which must create a linked process and return {ok, Pid}). Our kv:start_link/0 from the gen_server lesson fits.

ERLANG
1> kv_sup:start_link().
{ok,<0.150.0>}
2> exit(whereis(kv), kill).
true
3> kv:put(a, 1), kv:get(a).
{ok,1}

The server was killed, and the supervisor started a new one. (The state was lost: restarting gives a fresh init.)

Child specifications

Key Meaning
id Unique name inside this supervisor
start {M, F, A} that starts and links the child
restart permanent (always restart), transient (restart only after abnormal exit), temporary (never)
shutdown Milliseconds to wait after asking the child to stop, or brutal_kill, or infinity for a supervisor child
type worker or supervisor
modules Modules used for code upgrades (default [M])

All keys except id and start have defaults, so a short spec often suffices.

Restart strategies

  • one_for_one: if a child dies, only that child restarts. The right default for independent workers.
  • one_for_all: if one child dies, all children are terminated and restarted. Use when children depend on each other so strongly that a partial state is useless.
  • rest_for_one: the failed child and all children started after it (the later ones in the list) are restarted. Use it for ordered dependencies: the database connection first, the cache that uses it second.
  • simple_one_for_one (or the modern way: one_for_one with a template used by supervisor:start_child/2) for dynamic pools of identical workers, such as one per connection.

Children start in list order and are stopped in reverse order.

Restart intensity

Restarting forever would hide a bug that fails instantly. The intensity and period flags set a limit: at most intensity restarts in period seconds (defaults 1 restart in 5 seconds). When exceeded, the supervisor gives up: it terminates all its children and exits itself with reason shutdown. Its own supervisor then reacts, so the failure escalates upward until some level can repair it, or the whole application stops. That is the safety valve that keeps a crash loop from consuming a machine.

Restart strategies

one_for_one Restart only the failed child
one_for_all Restart every child
rest_for_one Restart it and later siblings
Dynamic pool Template, start_child on demand

Pick by how the children depend on each other.

1 / 2

Supervision trees

Because a supervisor is a process, it can be a child of another supervisor. A typical application has one top supervisor, with sub-supervisors per area:

flowchart TD
    A["app_sup (one_for_one)"] --> B["db_sup (rest_for_one)"]
    A --> C["web_sup (one_for_one)"]
    B --> D["db_pool"]
    B --> E["cache"]
    C --> F["listener"]
    C --> G["connection_sup (dynamic)"]
    G --> H["connection 1"]
    G --> I["connection 2"]

Workers do the work at the leaves; supervisors form the branches.

Design rule: put the code that can fail (parsing, I/O) in the leaves and the stable, high-value state near the root, and design restarts so they return to a known good state. Start-up should be able to rebuild anything it needs from outside the process: files, a database, other services.

Dynamic children

ERLANG
supervisor:start_child(Sup, ChildSpec)      % add a child at run time
supervisor:terminate_child(Sup, Id)
supervisor:delete_child(Sup, Id)
supervisor:which_children(Sup)              % [{Id, Pid, Type, Modules}]
supervisor:count_children(Sup)

For a pool of identical workers give the supervisor an empty child list and a template:

ERLANG
init([]) ->
    Flags = #{strategy => simple_one_for_one, intensity => 5, period => 10},
    Template = #{id => worker, start => {job_worker, start_link, []}, restart => temporary},
    {ok, {Flags, [Template]}}.
%% supervisor:start_child(Sup, [Arg1]) calls job_worker:start_link(Arg1)

Run it: a supervisor stand-in

This is a JavaScript model, not OTP. It restarts "children" according to a strategy and enforces intensity and period. Try changing the strategy to one_for_all or rest_for_one and observe which children restart, or lower the intensity to see escalation.

JS Stand-in: restart strategies

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Good practice

  • Keep supervisors free of logic; do risky work in workers.
  • Use transient for jobs that may finish normally, temporary for one-offs, permanent for services.
  • Order children by dependency, and pick rest_for_one if later children need earlier ones.
  • Make init in workers cheap and defer slow set-up so one slow child does not delay the siblings.
  • Use realistic intensity: a flapping dependency may need 5 in 10, not 1 in 5.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Under rest_for_one, what restarts when the second of four children crashes?

    rest_for_one restarts the failed child and every child started after it.
  2. 2. Which restart type means: restart only if the child exited abnormally?

    transient children are restarted after an abnormal exit but left alone after a normal one.
  3. 3. What happens when a supervisor exceeds its restart intensity?

    Giving up escalates the problem up the supervision tree.
  4. 4. What must a child start function do?

    The supervisor needs a linked process so that it receives exit signals; the function returns {ok, Pid}.

Summary

  • A supervisor starts, watches and restarts child processes using a list of child specs.
  • Strategies: one_for_one, one_for_all, rest_for_one, plus dynamic children.
  • intensity and period limit restarts and cause escalation up the tree.
  • Restart types: permanent, transient, temporary.
  • Build trees so that a restart always returns to a known good state.

Next: Applications and releases