Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 37 ยท Health Checks

Health States, Node Checks and Maintenance Mode

See how Consul combines checks into a service status, what serfHealth and node checks do, how to drain instances with consul maint, and how to alert on critical checks.

A single service instance can have several checks, and the machine it runs on has checks too. Consul has to boil all of them down to one answer to the question "may this instance receive traffic?". This lesson explains that aggregation, the built-in node-level checks, and maintenance mode, the tool for taking an instance out of rotation on purpose without stopping it.

The commands are read-only on this site (no Consul agent); the stand-ins model status aggregation and the drain procedure.

Combining checks

Each check is passing, warning or critical. The status of a service instance is the worst of all the checks that apply to it:

TEXT
critical > warning > passing

Which checks apply? The ones attached to that service instance and the node-level checks of the machine it runs on. So a service whose own HTTP check passes still counts as critical if the node's serfHealth check is failing.

Service checks Node checks Instance status
passing passing passing
warning passing warning
passing critical critical
critical warning critical

What each state does to lookups:

  • passing: returned by DNS and by ?passing.
  • warning: returned by DNS (unless only_passing), included by the health API but not by ?passing.
  • critical: never returned by DNS or by ?passing.

Node-level checks and serfHealth

Every agent registers a check called serfHealth. It is not a probe: it reflects the cluster's gossip view of the node. While gossip says the agent is alive, the check is passing; when the agent is marked failed or left, it turns critical, and every service on that node becomes unavailable at once. This is how Consul handles the case where a whole machine disappears and cannot report its own checks.

You can add your own node checks (not tied to a service) to describe the machine, for example disk space or a kernel setting:

HCL
check {
  name     = "disk space"
  args     = ["/usr/local/bin/check-disk.sh"]
  interval = "60s"
}

A critical node check takes down all services on that node, so use node checks for conditions that really disable the whole machine.

Querying states

Terminal
$ consul catalog nodes -detailed          # nodes with addresses and metadata
$ curl "http://127.0.0.1:8500/v1/health/state/critical"   # every critical check
$ curl "http://127.0.0.1:8500/v1/health/state/warning"
$ curl "http://127.0.0.1:8500/v1/health/state/any"
$ curl "http://127.0.0.1:8500/v1/health/node/web-1"       # all checks of one node
$ curl "http://127.0.0.1:8500/v1/health/checks/web"       # all checks of a service

The reply of a check includes Node, CheckID, Name, Status, Notes, Output (the last probe output, which is the first place to look when something fails), ServiceID and ServiceName.

Maintenance mode

Sometimes an instance is perfectly alive but you want no new traffic: during a deploy, a database migration, or a debugging session. Stopping it abruptly drops requests in flight. Maintenance mode registers a special critical check so the instance disappears from healthy lookups while it keeps running and finishing its work.

Terminal
# take the whole node out of rotation
$ consul maint -enable -reason "kernel patching"
$ consul maint                          # show what is in maintenance
$ consul maint -disable

# take only one service on this node out
$ consul maint -enable -service web -reason "deploying 2.5"
$ consul maint -disable -service web

The API equivalents are PUT /v1/agent/maintenance?enable=true&reason=... for the node and PUT /v1/agent/service/maintenance/<service-id>?enable=true&reason=... for one service. Internally Consul adds a check named _node_maintenance or _service_maintenance:<id> in the critical state, with your reason in its notes, which is why it shows in the UI and in health/state/critical. Maintenance mode survives agent restarts, so a rebooted machine does not rejoin the rotation until you disable it.

A safe deployment drain

sequenceDiagram
  participant Ops as Deploy script
  participant A as Consul agent
  participant LB as Clients / DNS
  Ops->>A: maint -enable -service web
  A-->>LB: web instance no longer returned
  Note over Ops: wait for in-flight requests (drain period)
  Ops->>Ops: stop, upgrade and start web
  Ops->>A: maint -disable -service web
  A-->>LB: web returns once its checks pass

Drain, upgrade, then re-enable; at no point does a client get a dead address.

The drain period must be longer than your DNS TTL and client cache times, otherwise some clients still hold the old address. Applications that register and deregister themselves can achieve the same by deregistering at the start of a graceful shutdown.

Leaving the cluster cleanly

A machine that stops without warning is marked failed by gossip and its services disappear after the failure is detected (and may linger in a failed state for a while). A graceful leave tells the cluster immediately:

Terminal
$ consul leave

Agents also leave on SIGINT (Ctrl+C) by default, and you can set leave_on_terminate = true so a SIGTERM from systemd or an orchestrator does the same. Servers that leave reduce the Raft quorum, so for servers prefer replacing a node by joining the new one first and removing the old one afterwards, never leaving below a majority.

Getting alerts

You can feed critical checks into your alerting without extra tools:

Terminal
$ consul watch -type=checks -state=critical ./notify-chat.sh

The handler receives the JSON list of critical checks on standard input. Prometheus users instead scrape agent metrics; commercial users use the UI's notifications. Whatever you choose, alert on duration (critical for more than N minutes), not on every blip.

Stand-in 1: aggregate instance status

Combines node-level and service-level checks with the worst-wins rule and decides which lookups would return the instance. Invented data.

Python Stand-in: worst check wins

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in 2: a drain simulation

A load of requests arrives each second while the instance is drained. The model shows why waiting before the upgrade matters: clients with cached answers keep sending traffic for as long as their TTL.

Python Stand-in: drain and TTL

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The second program is a simplified model of cache timing, not a measurement of any real client. Real runtimes cache DNS in different ways, so test yours.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A service's own HTTP check passes but its node's serfHealth check is critical. What status does the instance have?

    Node checks apply to every service on the node, and the worst state wins.
  2. 2. What does consul maint -enable -service web do?

    Maintenance mode adds a critical check, draining traffic without stopping the application.
  3. 3. What is serfHealth based on?

    serfHealth turns critical when gossip marks the agent as failed or left.
  4. 4. Why wait before stopping a drained instance?

    The drain period should exceed DNS TTLs and cover requests already in progress.

Summary

  • Instance status is the worst of its service checks and its node checks (critical beats warning beats passing).
  • serfHealth reflects gossip membership; a failed node takes all its services out at once.
  • consul maint (node or service) drains traffic by adding a critical check and survives restarts.
  • Drain longer than DNS TTLs and client caches before stopping an instance; leave gracefully with consul leave.
  • Alert on how long a check stays critical, for example with consul watch -type=checks.

Next: Designing Health Endpoints and Check Patterns