Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 37 ยท Health Checks

Designing Health Endpoints and Check Patterns

Design health endpoints that work with Consul: liveness versus readiness, shallow versus deep checks, using 429 for degraded states, avoiding cascading failures and startup and shutdown flows.

You now know every knob on a Consul check. The harder skill is deciding what a check should test. A check that is too shallow keeps broken instances in rotation; a check that is too deep can pull every instance out of rotation at once when a shared dependency hiccups, turning a small problem into a full outage. This lesson collects the design patterns that work in practice.

The Consul parts are read-only HCL; the stand-ins are runnable Python models of health endpoints and of a cascading failure.

Three questions a check can answer

Kind Question Consequence of "no"
Liveness Is the process running and not stuck? Restart it (orchestrator)
Readiness Can it accept traffic right now? Remove from rotation (Consul check)
Dependency health Are the things it needs working? Report it; usually degrade, not remove

Consul checks implement readiness: the answer decides whether a caller is sent to this instance. Liveness belongs to your process supervisor or orchestrator (systemd, Kubernetes). Do not use a Consul check to restart things; mixing the two causes restart loops during dependency outages.

Shallow and deep checks

  • A shallow check only confirms the process answers (GET /health returns 200 immediately). It is cheap and rarely fails for the wrong reason, but it cannot see a lost database connection.
  • A deep check also verifies critical dependencies (a database ping, a queue connection). It is informative, but it multiplies load and couples every instance to the dependency.

A balanced recipe:

  1. Keep the main readiness check cheap: local state only, answered within milliseconds.
  2. Include dependencies only when the service is useless without them, and test them with a cached, rate-limited result (for example "the last successful query was under 10 seconds ago"), not a fresh query per probe.
  3. Report optional dependencies (a recommendation engine, a cache) as warning, not critical.

Using HTTP status codes as signals

Because Consul maps HTTP 2xx to passing and 429 to warning (see check types), a health endpoint can express three states without a script:

Endpoint returns Meaning Consul state
200 Fully healthy passing
429 Degraded: working but limited (cache down, replica lag) warning
503 Cannot serve critical

With only_passing = false (the default for DNS), warning instances stay in DNS answers but are excluded when a client asks for ?passing; services that must tolerate a degraded peer can decide for themselves which they accept.

The cascading failure trap

Imagine ten web instances that all check the same database in their health endpoint. The database has a 20-second hiccup. All ten checks go critical together, Consul returns no instances, and the whole service is unreachable, even though the database recovered after 20 seconds and your servers were fine the whole time. Meanwhile the traffic that retries hammers the recovering database.

Defenses:

  • Shallow readiness plus separate dependency monitoring.
  • failures_before_critical = 3 or higher so short blips pass unnoticed (see check timing).
  • Serve stale or degraded responses when dependencies fail, and say so through warning.
  • Clients should retry across instances and use timeouts and circuit breakers; discovery is not a substitute for resilience.

Startup and shutdown

Startup. Register the service early but keep the check failing (initial_status = "critical", the default) until the app has loaded caches, opened pools and warmed up. Return 503 from /health until ready.

Shutdown. Graceful shutdown has an order:

  1. Flip /health to fail, or enter maintenance mode, or deregister.
  2. Wait a drain period longer than client DNS TTLs.
  3. Stop accepting connections, finish in-flight requests.
  4. Exit; the agent removes or fails the registration.

Orchestrated containers often do this in a preStop hook or a signal handler.

Several checks per service

A service may carry several independent checks, and its status is the worst of them:

HCL
service {
  name = "payments"
  port = 8443

  checks = [
    {
      name     = "http ready"
      http     = "http://127.0.0.1:8443/ready"
      interval = "10s"
      timeout  = "2s"
    },
    {
      name     = "tcp listener"
      tcp      = "127.0.0.1:8443"
      interval = "10s"
    },
    {
      name = "queue consumer heartbeat"
      ttl  = "45s"
    }
  ]
}

Name checks clearly: the check name and its last Output are what an engineer sees at 3 a.m. in the UI. Return concise reasons in the response body (for example db: connection refused), because Consul stores the output.

Common mistakes

  • Checking through the load balancer. Probe the instance directly, not through a path that could hit a different instance.
  • Authentication on the health endpoint. The agent needs a simple unauthenticated local call (bind it to localhost) or a static header.
  • Expensive probes. A health endpoint that runs a heavy SQL query every 5 seconds on 200 instances is a self-inflicted load test.
  • Same check for liveness and readiness, causing restarts during dependency outages.
  • No timeout awareness. A timeout shorter than a normal garbage-collection pause makes healthy services look dead.

Stand-in 1: a health endpoint with degrade rules

A function that turns dependency results into an HTTP status the way the table above recommends, with required and optional dependencies. Then it prints how Consul would interpret each response.

Python Stand-in: health endpoint logic

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in 2: cascading failure, shallow versus deep

Ten instances, one shared database that is down for 20 seconds out of a 60-second run. The deep health check removes every instance for the whole outage plus recovery lag; the shallow one keeps serving (cached or degraded responses) throughout. A small, deterministic model.

Python Stand-in: deep checks and outages

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The second program is a deliberately simple model that shows the shape of the problem. In a real system the probes of different instances are offset from each other and recovery takes time, so use it for intuition, not for measuring.

A review checklist

  • Does the readiness check finish in milliseconds and avoid heavy work?
  • Which dependencies are truly required, and which only degrade the service?
  • Do I return 503, 429 and 200 deliberately?
  • Do failures_before_critical, interval and DNS TTL together match my recovery goals?
  • Is there a drain period on shutdown and deploys?
  • Do checks have clear names and useful output?

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which question should a Consul health check mainly answer?

    Consul decides where traffic goes, so checks express readiness. Restarts belong to the supervisor or orchestrator.
  2. 2. What status should a health endpoint return when only an optional cache is down?

    429 maps to warning: the instance works but is degraded.
  3. 3. Why can deep dependency checks cause a total outage?

    Coupling all instances to one dependency means one failure removes all of them.
  4. 4. What is a safe shutdown order for an instance?

    Drain first so clients stop selecting the instance before it disappears.

Summary

  • Consul checks express readiness; leave liveness and restarts to your supervisor.
  • Keep readiness checks cheap and mostly local; include only required dependencies, cached.
  • Use 200, 429 and 503 to give passing, warning and critical without scripts.
  • Avoid cascading failures: shallow checks, failures_before_critical, degraded modes and client retries.
  • Drain before shutdown, start critical until ready, and give checks clear names and output.

Next: Service Mesh Concepts with Consul