Skip to content
Tutorials Center
My learning

Topics

Lesson 98 of 111 ยท Orchestration Basics

Swarm Services, Stacks and Rolling Updates

Go deeper into Docker Swarm: service modes, stack files with deploy settings, rolling updates and rollbacks, secrets, configs, and how to rotate them safely.

Docker Swarm Basics showed you the vocabulary: nodes, services, tasks and stacks. This lesson is about the part you will touch every day once a swarm exists, which is shipping changes: describing a whole application in one file, updating it without downtime, rolling it back when a release is bad, and delivering passwords and configuration files without baking them into images.

You will meet the settings that control a rolling update, the different service modes, the way secrets and configs are stored and mounted, and the one trap that makes people think their config change "did nothing". Everything is built around one idea you already know: you declare the desired state, and Swarm works until reality matches it.

A swarm needs several machines (or at least a daemon switched into swarm mode), and the daemon used to verify this track is shared, so the swarm commands below are read-only examples whose output follows how Swarm behaves. What was run for real: the stack file in this lesson, validated with docker stack config, and the images it uses. Treat every docker service and docker stack deploy block as illustrative.

Services have a mode

Every service runs in one of a few modes, chosen at creation time and fixed afterwards:

Mode Flag What Swarm does
Replicated --mode replicated (default) Runs the number of tasks you ask for, wherever there is room
Global --mode global Runs exactly one task on every eligible node, including nodes that join later
Replicated job --mode replicated-job Runs N tasks to completion, then stops (a batch run)
Global job --mode global-job Runs one task to completion on each node

Replicated is right for web servers and APIs. Global suits per-machine agents: a log shipper, a metrics exporter, a security scanner. The job modes are for work that should finish and not be restarted, such as a migration or a one-off cleanup on each node.

CONSOLE
$ docker service create --name agent --mode global --network host \
    --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock,ro \
    example/node-agent:1.4

(Read-only example.) You control where replicated tasks land with constraints and preferences. A constraint is a hard rule, a placement preference only nudges:

CONSOLE
$ docker service create --name web --replicas 6 \
    --constraint 'node.role == worker' \
    --placement-pref 'spread=node.labels.zone' \
    --replicas-max-per-node 2 \
    nginx:1.29-alpine

Here Swarm only uses workers, spreads the six tasks evenly over the values of the node label zone, and never puts more than two on one node. If the rules cannot be met, tasks stay Pending and docker service ps --no-trunc web explains why.

The life of a task

A service is a list of tasks, and each task walks through states. You will see them in the CURRENT STATE column of docker service ps.

stateDiagram-v2
    [*] --> New
    New --> Pending: waiting for a node
    Pending --> Assigned: scheduler picked a node
    Assigned --> Preparing: pulling the image
    Preparing --> Starting: container created
    Starting --> Running: healthcheck passes
    Running --> Complete: exited with 0 (job)
    Running --> Failed: exited with an error
    Running --> Shutdown: replaced or scaled down
    Preparing --> Rejected: image pull failed

A task is never restarted in place: when it fails, the orchestrator creates a new task and the old one stays in the history.

The history is useful. docker service ps web lists the current tasks and the dead ones, with an ERROR column such as "task: non-zero exit (1)" or "No such image: ...". How many old tasks are kept is a swarm setting (docker swarm update --task-history-limit, default 5 per slot).

One file for the whole application: stacks

A stack is a set of services, networks, volumes, secrets and configs deployed from a single Compose-format file. The deploy: key under each service carries the Swarm-only settings. This file was checked with the real docker stack config:

YAML
services:
  web:
    image: nginx:1.29-alpine
    ports:
      - "8080:80"
    configs:
      - source: site_conf
        target: /etc/nginx/conf.d/default.conf
    networks:
      - front
    deploy:
      replicas: 3
      update_config:
        parallelism: 1
        delay: 10s
        order: start-first
        failure_action: rollback
        monitor: 20s
      rollback_config:
        parallelism: 0
        order: stop-first
      restart_policy:
        condition: on-failure
        max_attempts: 3
      resources:
        limits:
          cpus: "0.5"
          memory: 128M
        reservations:
          memory: 64M
  db:
    image: postgres:17-alpine
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password
    volumes:
      - dbdata:/var/lib/postgresql/data
    networks:
      - back
    deploy:
      placement:
        constraints:
          - node.labels.disk == ssd
configs:
  site_conf:
    file: ./site.conf
secrets:
  db_password:
    file: ./db_password.txt
volumes:
  dbdata:
networks:
  front:
  back:

docker stack config -c stack.yaml normalizes it exactly like a deploy would, and it needs no swarm. The interesting part of its real output shows what Swarm will actually receive:

CONSOLE
$ docker stack config -c stack.yaml
...
      resources:
        limits:
          cpus: "0.5"
          memory: "134217728"
        reservations:
          memory: "67108864"
...
    ports:
      - mode: ingress
        target: 80
        published: 8080
        protocol: tcp

Memory sizes became bytes, and the short port syntax became an ingress port (published through the routing mesh, explained in Swarm Nodes, Overlay Routing Mesh and Managing Clusters). Run this command in CI to catch a typo before a deploy does.

Deploying and inspecting (read-only example):

CONSOLE
$ docker stack deploy -c stack.yaml shop --with-registry-auth
Creating network shop_front
Creating network shop_back
Creating secret shop_db_password
Creating config shop_site_conf
Creating service shop_db
Creating service shop_web

$ docker stack ls
NAME   SERVICES
shop   2

$ docker stack ps shop
ID             NAME         IMAGE                NODE    DESIRED STATE   CURRENT STATE
7q1x2y3z4a5b   shop_web.1   nginx:1.29-alpine    node1   Running         Running 20 seconds ago
8r2y3z4a5b6c   shop_web.2   nginx:1.29-alpine    node2   Running         Running 20 seconds ago
9s3z4a5b6c7d   shop_web.3   nginx:1.29-alpine    node3   Running         Running 19 seconds ago
0t4a5b6c7d8e   shop_db.1    postgres:17-alpine   node2   Running         Running 24 seconds ago

Four rules explain most stack surprises:

  • Names get the stack prefix. web becomes shop_web, and the networks, secrets and configs get it too. The services still reach each other by their short name (db) on the stack's networks.
  • Deploy again to update. Edit the file and run the same docker stack deploy. Swarm compares each service with the file and only touches what changed.
  • Removed services stay until you prune. Deleting a service from the file does nothing by default; docker stack deploy --prune removes services that are no longer in it.
  • Not everything in Compose applies. build: is ignored (build and push to a registry first), and depends_on, container_name and restart: do nothing. Use deploy.restart_policy, healthchecks and retry logic in your application. A Swarm stack also does not read a .env file the way docker compose does: variables in ${...} come from the shell that runs the deploy.

Rolling updates in detail

When you change a service's image, command, environment or limits, Swarm does not touch all tasks at once. It performs a rolling update governed by update_config.

sequenceDiagram
    participant Op as You
    participant S as Swarm
    participant Old as web.1 (v1)
    participant New as web.1 (v2)
    Op->>S: docker service update --image v2
    S->>New: start new task (order start-first)
    New-->>S: healthcheck passes, monitor period ok
    S->>Old: stop old task
    Note over S: wait delay, then next batch of parallelism tasks
    S-->>Op: service converged

With start-first the new task is healthy before the old one is stopped, so capacity never dips.

The settings, in the order you usually think about them:

Setting Meaning
parallelism How many tasks update at the same time. 0 means all at once.
delay Pause between batches.
order stop-first (default: free the resources, then start) or start-first (start the new task, then stop the old).
monitor How long a new task must stay healthy before it counts as a success (default 5s).
failure_action What to do if a task fails: pause (default), continue or rollback.
max_failure_ratio Fraction of failing tasks you tolerate before failure_action triggers.

start-first needs spare capacity because old and new tasks run side by side for a moment, and it is not safe for a service that cannot run two copies (a database on one volume). Use stop-first there.

The CLI flags mirror these names, which is handy for a one-off change without editing a file:

CONSOLE
$ docker service update \
    --update-parallelism 1 --update-delay 10s --update-order start-first \
    --update-failure-action rollback --update-monitor 20s \
    --image nginx:1.30-alpine shop_web
shop_web
overall progress: 3 out of 3 tasks
1/3: running   [==================================================>]
2/3: running   [==================================================>]
3/3: running   [==================================================>]
verify: Service shop_web converged

(Read-only example.) While it runs, docker service inspect --pretty shop_web shows UpdateStatus: State: updating, and when it ends completed, paused (a failure with failure_action: pause) or rollback_completed.

Rollbacks

rollback_config mirrors update_config for the way back. If an update fails and failure_action is rollback, Swarm restores the previous service definition, one batch at a time. You can also do it by hand:

CONSOLE
$ docker service rollback shop_web

Swarm only remembers one previous definition, so a rollback is one step back, not a history. For anything older, deploy the old tag again. A related fact: when you pass a tag like nginx:1.30-alpine, Swarm resolves it to the image digest on the manager and pins the tasks to that digest, so every node runs byte-identical images even if the tag moves later. --no-resolve-image turns this off (for registries the manager cannot reach).

Updates need healthchecks

Without a HEALTHCHECK Swarm believes a task is healthy the moment its main process starts. A web server that listens only after 20 seconds of warm-up would be counted as ready immediately, and the update would roll on and replace every healthy copy with one that is still booting. Define a healthcheck in the image or in the service (healthcheck: in the stack file), and let monitor cover the first few seconds after it passes.

To redeploy the same definition (for example to make tasks re-pull a moved tag or spread over a changed set of nodes), use docker service update --force shop_web.

Secrets: sensitive values, delivered safely

A Swarm secret is a blob (up to 500 KB) stored encrypted in the managers' Raft log. Swarm sends it over mutual TLS only to the nodes that run a task needing it, and mounts it in the container as a file on an in-memory filesystem, /run/secrets/<name>. It is never written to the node's disk and never appears in docker inspect output or in the image. That is why the Postgres example above reads POSTGRES_PASSWORD_FILE instead of a password in an environment variable (see Managing Secrets for why environment variables are weaker).

CONSOLE
$ printf 's3cr3t-value' | docker secret create db_password_v1 -
$ docker secret ls
ID                          NAME             DRIVER    CREATED         UPDATED
k2m3n4p5q6r7s8t9u0v1w2x3y   db_password_v1             5 seconds ago   5 seconds ago
$ docker service create --name db \
    --secret source=db_password_v1,target=db_password,mode=0400 \
    -e POSTGRES_PASSWORD_FILE=/run/secrets/db_password \
    postgres:17-alpine

(Read-only example.) The source is the secret object, the target is the file name inside the container, and mode sets the permissions (the default is 0444; the container user must still be able to read it).

Secrets are immutable. You cannot edit one, so you rotate it by adding a new object and swapping it in one service update:

CONSOLE
$ printf 'n3w-s3cr3t' | docker secret create db_password_v2 -
$ docker service update \
    --secret-rm db_password_v1 \
    --secret-add source=db_password_v2,target=db_password,mode=0400 db

The file path inside the container stays /run/secrets/db_password, and the tasks are replaced one at a time. Afterwards you can docker secret rm db_password_v1. In a stack file, a secret created ahead of time is referenced with external: true; one declared with file: is created from that file at deploy time (convenient, but the file then sits on the machine you deploy from).

Configs: the same idea for non-sensitive files

A config works like a secret with two differences: it is not encrypted at rest, and it is mounted as a normal file wherever you point it (the default is /<name>). Use it for nginx.conf, prometheus.yml or an application settings file, so you do not have to rebuild an image to change one line.

CONSOLE
$ docker config create site_conf_v1 ./site.conf
$ docker service update \
    --config-add source=site_conf_v1,target=/etc/nginx/conf.d/default.conf web

Configs are immutable too, and this is where the classic trap lives. In a stack file, site_conf is created as shop_site_conf on the first deploy. If you edit site.conf and run docker stack deploy again, Swarm refuses with an error like only updates to Labels are allowed, because the object with that name already exists with different content. The fix is to put a version in the object name so each content change makes a new object, and to let the stack variable decide it:

YAML
configs:
  site_conf:
    name: site_conf_${SITE_CONF_VERSION:-1}
    file: ./site.conf
CONSOLE
$ SITE_CONF_VERSION=2 docker stack deploy -c stack.yaml shop
Creating config site_conf_2
Updating service shop_web (id: n2v8s0q1x7t3d5e6f9g0h1j2k)

Changing the config source in the service triggers a normal rolling update, with all the safeguards above. Some teams use a hash of the file instead of a number, so the name changes exactly when the content does.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. You want a log-shipping agent to run on every node, including nodes added later. Which mode fits?

    Global mode schedules one task per eligible node automatically, so new nodes get the agent without any change to the service.
  2. 2. A rolling update replaces tasks one at a time with order: start-first. What is the benefit compared with the default stop-first?

    With start-first, old and new tasks briefly run side by side. It needs spare resources and is wrong for services that cannot run twice, such as a database on one volume.
  3. 3. Why do rolling updates become unsafe if the image has no HEALTHCHECK?

    Healthchecks tell Swarm when a task is really ready. Without one, readiness equals 'the process exists'.
  4. 4. You edit site.conf, which a stack file mounts as a config, and run docker stack deploy again. It fails because the config already exists with other content. What is the standard fix?

    Configs and secrets are immutable. A new name (site_conf_2, or a content hash) creates a new object, and the service update swaps it in with a rolling update.
  5. 5. Where does a Swarm secret appear inside the container?

    Secrets are mounted as files in tmpfs, so they are not written to the node's disk and do not show up in image layers or docker inspect.

Summary

  • Services run in replicated, global or job modes; constraints, preferences and replicas-max-per-node control placement.
  • A stack deploys services, networks, volumes, secrets and configs from one Compose-format file; the deploy: key holds the Swarm settings, and docker stack config validates it without a swarm.
  • Rolling updates are tuned with parallelism, delay, order, monitor, failure_action and max_failure_ratio; rollback_config controls the way back, and docker service rollback returns one step.
  • Healthchecks are what make updates safe, and Swarm pins tasks to the image digest it resolved.
  • Secrets and configs are immutable objects: rotate by creating a new, preferably versioned, object and swapping it in with one service update. Secrets reach containers as in-memory files under /run/secrets/.
  • build:, depends_on and .env files do not work like they do with docker compose.

Next: Swarm Nodes, Overlay Routing Mesh and Managing Clusters