Skip to content
Tutorials Center
My learning

Topics

Lesson 99 of 111 ยท Orchestration Basics

Swarm Nodes, Overlay Routing Mesh and Managing Clusters

Manage a Docker Swarm cluster: manager quorum, promoting and draining nodes, labels, autolock, backups, the ingress routing mesh and overlay networks.

A swarm is only as dependable as the machines that make it up. Services and stacks (Swarm Services, Stacks and Rolling Updates) describe what you want; the nodes are where it happens, and the network is what lets users reach it. This lesson covers the cluster itself: how managers agree on state, how to add, promote, drain and remove nodes, how to protect and back up the cluster, and how traffic flows through the ingress routing mesh and overlay networks.

Most of the commands here are things you run once a month, not once a day, which is exactly why they are worth reading now: the day a node dies is a bad moment to learn them.

Swarm needs several machines, and the Docker daemon used to verify this track is shared and not in swarm mode, so every swarm command in this lesson is a read-only example written from how Swarm behaves. Substitute your own node names and addresses (192.0.2.x is a reserved documentation range).

Managers, workers and quorum

Every node is a manager or a worker. Workers only run tasks. Managers also keep the cluster state (services, secrets, configs, node list) in a replicated log, take your commands and run the scheduler. Managers agree on every change using the Raft consensus algorithm, one of them acting as the leader; a change is accepted only when a majority (quorum) of managers has stored it.

Managers Majority needed Failures tolerated
1 1 0
2 2 0
3 2 1
4 3 1
5 3 2
7 4 3

Read the table twice. Two managers are worse than one (either failing stops the cluster), and four are no better than three. Use 1 for a lab, 3 for most real clusters and 5 when you must survive two failures; Docker advises against more than 7 because every write must reach a majority, so each extra manager slows the cluster down. Spread managers over different machines, racks or zones, otherwise one failure takes out several of them at once.

What happens when quorum is lost? The tasks that are already running keep running. But nothing can change: you cannot create or update services, and Swarm cannot reschedule tasks from a node that dies. Restoring a majority of managers fixes it.

flowchart TB
    subgraph managers["Managers (Raft quorum)"]
        L["manager1<br/>leader"]
        M2["manager2"]
        M3["manager3"]
        L <--> M2
        M2 <--> M3
        L <--> M3
    end
    W1["worker1<br/>tasks"]
    W2["worker2<br/>tasks"]
    W3["worker3<br/>tasks"]
    managers -->|"orders and state"| W1
    managers -->|"orders and state"| W2
    managers -->|"orders and state"| W3

Three managers keep the shared state; workers receive tasks and report back.

By default managers also run tasks. On a bigger cluster, keep managers free of application load so a busy service cannot starve the scheduler:

CONSOLE
$ docker node update --availability drain manager1
manager1

Adding, listing and promoting nodes

A swarm starts with docker swarm init on one machine. Other machines join with a token, which is a secret: whoever has the worker token can add a machine to your cluster.

CONSOLE
$ docker swarm init --advertise-addr 192.0.2.10
Swarm initialized: current node (x9k2d1m4q7ab) is now a manager.

$ docker swarm join-token worker
To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-<secret>-<secret> 192.0.2.10:2377

$ docker swarm join-token manager      # a different token, for managers
$ docker swarm join-token --rotate worker   # invalidate the old worker token

Use --advertise-addr whenever a machine has more than one network interface, and --listen-addr to choose where the manager API listens. A node must be able to reach the others on 2377/tcp (cluster management), 7946/tcp and udp (node discovery) and 4789/udp (overlay traffic). Keep these ports closed to the internet.

On a manager you see and change the nodes:

CONSOLE
$ docker node ls
ID                  HOSTNAME   STATUS   AVAILABILITY   MANAGER STATUS   ENGINE VERSION
x9k2d1m4q7ab *      node1      Ready    Active         Leader           29.8.1
p3t8w0c5n2ef        node2      Ready    Active         Reachable        29.8.1
k7r1y6h9s4gh        node3      Ready    Active         Reachable        29.8.1
b5v9c2x6z1lm        node4      Ready    Active                          29.8.1

MANAGER STATUS is empty for workers, Leader for the one that coordinates, and Reachable for the others. A manager that cannot be contacted shows Unreachable. STATUS is the health of the node (Ready or Down), and AVAILABILITY is what you decide:

Availability Meaning
active The scheduler may place new tasks here.
pause Existing tasks stay, but no new tasks are placed here.
drain No new tasks, and existing tasks are moved to other nodes.

Promote a worker when you need another manager and demote it again when you need fewer:

CONSOLE
$ docker node promote node4
Node node4 promoted to a manager in the swarm.
$ docker node demote node4
Manager node4 demoted in the swarm.

Change managers one at a time and check docker node ls between steps, so you never drop below quorum by accident.

Maintenance: drain, update, return

The safe way to patch or reboot a machine is to drain it first. Swarm stops its tasks and starts replacements elsewhere, honoring each service's update_config.

CONSOLE
$ docker node update --availability drain node3
node3
$ docker service ps shop_web
ID             NAME             NODE    DESIRED STATE   CURRENT STATE
a1b2c3d4e5f6   shop_web.3       node4   Running         Running 5 seconds ago
f6e5d4c3b2a1    \_ shop_web.3   node3   Shutdown        Shutdown 6 seconds ago

After the maintenance:

CONSOLE
$ docker node update --availability active node3
node3

Be aware that Swarm does not rebalance when a node returns. Tasks that moved away stay where they are, and the returned node gets work only when new tasks appear (a scale-up, an update, or docker service update --force).

To remove a node for good, make it leave and then delete its record:

CONSOLE
$ docker swarm leave                 # run on the node itself
Node left the swarm.
$ docker node rm node3               # run on a manager, once its status is Down
node3

Demote a manager to a worker before it leaves. docker swarm leave --force on the last manager of a swarm destroys the cluster state.

Labels and roles

Labels are free-form key/value tags on a node. You use them in placement constraints, as in the previous lesson:

CONSOLE
$ docker node update --label-add disk=ssd --label-add zone=eu-1 node2
node2
$ docker node inspect --format '{{ .Spec.Labels }}' node2
map[disk:ssd zone:eu-1]

Built-in attributes work in constraints too: node.role, node.hostname, node.id and node.platform.os. Prefer labels for properties you choose (hardware, location, ownership) and node.role == manager for control-plane services.

Protecting the cluster

Swarm already protects node-to-node traffic with mutual TLS and rotates node certificates (every 90 days by default). Three more habits are worth having.

Autolock. The managers' Raft log, including the encrypted secrets, is stored on disk, and the key that encrypts it sits next to it. With --autolock, Swarm encrypts that key as well, and a restarted manager stays locked until you supply the unlock key by hand:

CONSOLE
$ docker swarm init --autolock
...
To unlock a swarm manager after it restarts, run the `docker swarm unlock`
command and provide the following key:

    SWMKEY-1-<secret>

$ docker swarm unlock-key            # show it again later
$ docker swarm unlock                # after a manager restarts; prompts for the key

Store the key in a password manager, away from the cluster. Losing both it and the managers' state means losing the cluster.

Backups. Swarm's state lives in /var/lib/docker/swarm on each manager. A backup is: stop Docker on one manager (the cluster keeps quorum when there are three), archive that directory, start Docker again. You restore it on a fresh machine and re-initialize with docker swarm init --force-new-cluster, which makes that node the sole manager of a cluster with the old services and secrets still defined; then join the other nodes again. Back up your stack files in Git too: they are the most readable form of your desired state.

Token and certificate hygiene. Rotate join tokens after anyone who knew them leaves (docker swarm join-token --rotate), and if you suspect a node was compromised, docker swarm ca --rotate issues new certificates for the whole cluster.

The overlay network

Containers on different nodes need to talk by name. A Swarm overlay network is a virtual network that spans all nodes, built by wrapping container traffic in VXLAN packets (UDP 4789) between the machines. It is created on managers and appears on a worker only when a task of a service attached to it runs there.

CONSOLE
$ docker network create --driver overlay --attachable appnet
$ docker network create --driver overlay --opt encrypted secure-net

--attachable lets stand-alone containers (docker run --network appnet) join, which helps with debugging. --opt encrypted encrypts the VXLAN traffic with IPsec; it costs some CPU and throughput, so use it for networks that cross untrusted links. Inside an overlay network a service is found by name through Swarm's built-in DNS, which has two modes:

  • vip (default): the name resolves to one virtual IP. Swarm load-balances connections from that VIP across the healthy tasks.
  • dnsrr: the name resolves to the list of task IPs, in rotating order. You get no VIP, so the client does the balancing and may cache a dead address. Use it for software with its own client-side balancing, or if you must publish ports in host mode.

You set the mode per service with --endpoint-mode vip|dnsrr. Compare these with the single-host drivers in Network Drivers.

The ingress routing mesh

When you publish a port for a Swarm service, the default is ingress mode. Swarm opens that port on every node of the cluster, whether or not the node runs a task of the service, and forwards connections to a healthy task, wherever it runs. This is the routing mesh.

flowchart LR
    C(["Client"]) -->|"port 8080"| N2["node2<br/>no web task"]
    N2 -->|"ingress network<br/>(IPVS)"| T1["web.1<br/>on node1"]
    N2 -->|"ingress network<br/>(IPVS)"| T3["web.3<br/>on node3"]
    N1["node1"] --- T1
    N3["node3"] --- T3

A request that hits a node without a task is forwarded over the ingress network to a node that has one.

CONSOLE
$ docker service create --name web --replicas 2 --publish published=8080,target=80 nginx:alpine
$ curl -s -o /dev/null -w '%{http_code}\n' http://192.0.2.12:8080/    # node2 runs no task
200

Consequences for real deployments:

  • Put a load balancer or DNS record in front of all nodes (or of the nodes you choose). Any node accepts the connection, so there is no special entry node to protect or to fail.
  • Open the published port on every node's firewall, not only the ones running tasks.
  • The mesh load-balances connections, not individual requests. A long-lived HTTP/2 or WebSocket connection stays on the task it first reached.
  • With ingress mode the container does not see the real client IP (it sees the mesh's address), because the traffic is translated on the way. If you need it (for logs, rate limits, geo rules), publish in host mode:
CONSOLE
$ docker service create --name edge --mode global \
    --publish published=80,target=80,mode=host nginx:alpine

In host mode the port is opened only on the node that runs the task, there is no mesh and no extra hop, and the client address is preserved. Combine it with --mode global so each node has exactly one task, then let an external load balancer decide where to send traffic. Reverse proxies like those in Project: Multiple Apps Behind a Proxy are usually run this way in a swarm.

A last detail: the mesh uses a special network called ingress and a per-node bridge docker_gwbridge. Their default address ranges can clash with your corporate networks; choose different pools with docker swarm init --default-addr-pool 10.20.0.0/16 before you create the cluster.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A swarm has 3 managers and 2 of them fail. What happens?

    Raft needs a majority of managers (2 of 3). With one left there is no quorum, so state cannot change, although existing containers keep running.
  2. 2. You need to reboot a worker for a kernel update without dropping capacity. What do you do first?

    Draining moves the tasks to other nodes (respecting the update settings) and stops new tasks from being placed on the node. You make it active again afterwards.
  3. 3. A service publishes port 8080 in the default ingress mode, and the service has tasks only on node1 and node3. A request to node2:8080 will...

    Ingress mode opens the published port on every node of the swarm and forwards connections to a healthy task.
  4. 4. Your application logs show the same internal address for every client of a swarm service. How can you get the real client IP?

    Ingress-mode traffic is translated by the routing mesh. Host mode publishes on the node running the task and keeps the client address.
  5. 5. What does docker swarm init --autolock add?

    Autolock protects the encryption key of the on-disk cluster state (including secrets). After a restart, docker swarm unlock supplies the key.

Summary

  • Managers keep the state with Raft and need a majority: use 1 (lab), 3 or 5 managers, never an even number, and keep them on separate machines.
  • Losing quorum freezes changes but does not stop running containers.
  • docker node ls/inspect/update/promote/demote/rm manage nodes; drain before maintenance, active afterwards (Swarm does not rebalance by itself); labels feed placement constraints.
  • Protect the cluster with firewalls on 2377, 7946 and 4789, token rotation, --autolock, and backups of /var/lib/docker/swarm plus your stack files.
  • Overlay networks span nodes (VXLAN), can be encrypted, and give name resolution by VIP or DNS round robin.
  • The ingress routing mesh publishes a port on every node and forwards to a healthy task; use host mode when you need client IPs or no extra hop.

Next: Project: Serve a Static Site