Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 38 ยท Projects and Practice

Project: A Zero-Downtime Release Process

Design a release process for Swarm: build once, deploy by tag, canary with a second service, watch health, roll back automatically, plus a runnable rollout simulator.

Rolling updates and health checks are tools; a release process is how a team uses them every day so that deployments are boring. In this project you design that process for a Swarm service: what CI does, how a release is promoted, how a canary limits risk, and how you roll back without drama.

Swarm cannot run on this site, so the commands are read-only and I show no output. The runnable blocks are Python stand-ins: a release pipeline simulator and a canary traffic splitter that you can modify.

Principles

  1. Build once, deploy many. The artefact that passed tests in CI is the exact image that reaches production, identified by a version tag or digest.
  2. Immutable tags. Never push a different image to an existing tag (Private Registries).
  3. Deploy is a Git change. The stack file (with TAG) lives in a repository; a deploy is a reviewed change or a pipeline run, so history shows who changed what.
  4. Backward-compatible steps. Version N and N+1 must run together for a short time during the rollout, so database and API changes happen in two phases.
  5. Automatic safety net. Health checks plus failure_action: rollback (Rolling Updates).

The pipeline

flowchart LR
  G["git push"] --> T["CI: tests"]
  T --> B["Build and push<br/>api:1.7.0"]
  B --> S["Deploy to staging"]
  S --> V["Smoke tests"]
  V --> A{"Approve?"}
  A --> C["Canary in production"]
  C --> F["Full rollout"]
  F --> M["Watch metrics"]

Each step is automated except the approval, which can also be automatic for low-risk services.

A pipeline job for the production step often looks like this (shell, read-only):

Terminal
set -euo pipefail
export TAG="$CI_COMMIT_TAG"
docker stack config -c stack.yml -c stack.prod.yml > rendered.yml
docker stack deploy -c rendered.yml --with-registry-auth --detach=false shop
docker service inspect --format '{{.UpdateStatus.State}}' shop_api

The pipeline connects to a manager through SSH or a remote Docker context (docker context create prod --docker "host=ssh://deploy@manager1"). Give the deployment user the least privileges you can; remember that Docker API access is close to root on the cluster.

Fail the pipeline if the update state is rollback_completed or paused, and notify the team. A rollback you did not notice is a silent outage of the new feature.

Canary releases

Swarm has no built-in traffic splitting, but you can build a simple canary with two services behind one proxy:

YAML
services:
  api:
    image: registry.example.com/shop/api:1.6.0
    deploy: { replicas: 9 }
  api-canary:
    image: registry.example.com/shop/api:1.7.0
    deploy: { replicas: 1 }

A proxy (Traefik or nginx) that load-balances across both service names sends roughly one request in ten to the canary. Watch error rate and latency of the canary for a fixed window. If it is healthy, update api to 1.7.0 with a normal rolling update and remove api-canary; if not, delete the canary with no user-wide impact. Keep the canary on its own metrics label so dashboards can compare versions.

Database changes without downtime

Use the expand and contract pattern:

  1. Expand: add the new column or table (nullable, with defaults). Old code ignores it.
  2. Deploy the new code that writes both and reads the new one.
  3. Backfill existing rows.
  4. Contract: a later release drops the old column once no running version needs it.

Never rename or drop in the same release as the code change that depends on it, because during a rolling update both versions are alive.

Rollback runbook

  • Automatic: a failed health check within monitor triggers rollback to the previous spec.
  • Manual: docker service rollback shop_api returns to the previous spec (one level only).
  • Deeper: redeploy the older tag from Git: TAG=1.6.0 docker stack deploy ....
  • Data: if a migration cannot be reversed, rollback does not undo it. That is why migrations are backward compatible.

After any rollback, write down what happened. Small, frequent releases make each rollback easy to reason about.

Stand-in: simulate a canary

Python Stand-in: split traffic between stable and canary

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python Stand-in: release pipeline with gates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common mistakes

  • Rebuilding the image for production instead of promoting the tested one.
  • Reusing a version tag.
  • Ignoring the pipeline's exit status when Swarm rolls back by itself.
  • Combining a destructive migration with the code that needs it.
  • A canary without separate metrics, so nobody can tell versions apart.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does 'build once, deploy many' mean?

    Promoting one artefact guarantees you ship what you tested.
  2. 2. Why use the expand and contract pattern for database changes?

    Both versions must work against the same schema during the rollout.
  3. 3. How can you do a simple canary in Swarm?

    Swarm has no traffic-splitting primitive; two services behind a proxy approximate it.
  4. 4. After docker service rollback, how do you return to a version two releases back?

    Swarm remembers only the previous spec, so deeper rollbacks use the Git history.

Summary

  • A good release process builds once, deploys immutable tags, and keeps the deploy definition in Git.
  • Health checks, start-first and failure_action: rollback give an automatic safety net; the pipeline must still notice rollbacks.
  • Canary releases can be approximated with a second small service behind the proxy.
  • Make database changes backward compatible with expand and contract.

Next: Project: A Swarm Capacity Planner