Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 38 ยท Swarm Architecture

The Orchestrator and Reconciliation

See the internal components of a Swarm manager (orchestrator, allocator, scheduler, dispatcher) and how the reconciliation loop turns services into running tasks.

You type one line, docker service create, and containers appear on several machines. Between the two sits a pipeline of components inside every manager. Knowing the pipeline makes debugging much easier: when a service is stuck, you can ask which stage it is stuck in.

The pipeline

When you create or change a service, the request reaches the leader's API and is stored in the cluster log. Then these components, all running on the leader, act in order:

  1. Orchestrator. Compares desired state (service spec) with actual state (existing tasks). If the service wants 3 replicas and 2 tasks exist, it creates a new task object in state new. It also removes tasks when you scale down and replaces failed ones.
  2. Allocator. Gives the new task its network resources: IP addresses from the overlay networks and VIPs, and published ports. After this the task is pending.
  3. Scheduler. Chooses a node for each pending task by filtering and ranking nodes (constraints, resources, platform, spread). The task becomes assigned.
  4. Dispatcher. Sends assigned tasks to the chosen worker (or manager) over the management channel and collects status reports. It also keeps heartbeats with nodes.
  5. Worker agent. On the node, the agent hands the task to the container runtime: pulls the image, prepares mounts and secrets, creates and starts the container, runs health checks and reports every state change back.
flowchart LR
  A["Service spec"] --> B["Orchestrator: create tasks"]
  B --> C["Allocator: IPs and ports"]
  C --> D["Scheduler: choose node"]
  D --> E["Dispatcher: send to node"]
  E --> F["Worker agent: run container"]
  F -->|"status"| E

Every stage reads and writes the shared state, so a new leader can resume from where the old one stopped.

Reconciliation: the loop that never ends

The orchestrator does not "start containers once". It reacts to events and compares state:

  • a task fails or its node goes Down,
  • you scale or update the service,
  • a node is drained or removed,
  • a constraint changes.

Each event triggers a comparison. If actual does not match desired, the orchestrator creates or removes tasks. This is why Swarm self-heals without any monitoring script from you.

Some key behaviours that fall out of this design:

  • Tasks are immutable. A task never changes image or settings. To update a service, the orchestrator creates new tasks and shuts down old ones (a rolling update, see Scaling and Updates).
  • Tasks are never moved. A running task stays on its node until it dies or the node is drained; then a new task is scheduled elsewhere.
  • History is kept. Old failed tasks stay in the list (default retention is 5 per slot) so you can see why they failed with docker service ps --no-trunc.
  • A node going Down does not instantly cause rescheduling: the dispatcher waits for missed heartbeats (a few seconds) before declaring the node unavailable, then the orchestrator replaces the node's tasks.

Where a service can get stuck

Symptom Likely stage What to check
REPLICAS 0/3, tasks pending Scheduler Constraints that match no node, not enough CPU or memory, platform mismatch. docker service ps shows "no suitable node" in the error column.
Tasks assigned or preparing for long Worker agent Image pull is slow or failing, secrets cannot be delivered.
Tasks starting then failed repeatedly Container The application crashes: read docker service logs.
Task rejected Worker agent The node could not start it (missing mount path, port in use).

Global and replicated orchestration

The orchestrator has a replicated mode (a fixed count) and a global mode (one per eligible node). See Replicated and global services. In both cases the same compare-and-fix principle applies: global mode compares "eligible nodes" with "tasks" instead of a number.

Stand-in: a mini orchestrator and scheduler

The next editor combines the stages in about 30 lines: it keeps desired state, creates tasks, assigns them to the least loaded node and reconciles after a node failure. It is a teaching model, not Swarm's code.

Python Stand-in: mini orchestrator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in: diagnose a stuck task

Practise reading the symptoms table. The function maps what you observe to the most probable stage.

Python Stand-in: where is it stuck?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 3 questions

of 3 answered You scored / 3 ยท perfect!

  1. 1. Which component decides on which node a task runs?

    The scheduler filters and ranks nodes for pending tasks; the dispatcher then delivers the assignment.
  2. 2. You update a service's image. What does the orchestrator do?

    Tasks are immutable; an update replaces tasks.
  3. 3. Tasks stay in 'pending' with the error 'no suitable node'. Where is the problem most likely?

    Pending means the scheduler has not found a node that satisfies the task's requirements.

Summary

  • Orchestrator, allocator, scheduler and dispatcher run on the leader; agents run on every node.
  • Reconciliation compares desired and actual state on every event.
  • Tasks are immutable and never move; failures create new tasks.
  • The task state tells you which stage is stuck.

Next: Service Create Options