Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 37 ยท Health Checks

Check Timing, TTLs and Avoiding Flapping

Tune Consul health checks: interval, timeout, success_before_passing, failures_before_critical, initial_status, TTL heartbeats and deregister_critical_service_after, with detection-time math.

A health check that reacts too slowly keeps sending traffic to a dead instance. One that reacts too quickly turns every garbage-collection pause into an outage and makes instances flap between passing and critical. Consul gives you a handful of timing settings to find the balance. This lesson explains each, shows how to estimate detection time, and walks through TTL heartbeats.

The settings are read-only HCL here; the stand-ins simulate the state machine and TTL expiry with a logical clock.

The timing settings

HCL
check {
  name     = "web http"
  http     = "http://127.0.0.1:8080/health"
  interval = "10s"
  timeout  = "3s"

  success_before_passing   = 2
  failures_before_critical = 3

  deregister_critical_service_after = "30m"
}
Setting Meaning
interval Time between probes (required for active checks)
timeout How long a single probe may take before counting as a failure (default 10s for HTTP and TCP)
success_before_passing Consecutive successes needed to move to passing
failures_before_warning Consecutive failures needed to enter warning (when the probe reports it)
failures_before_critical Consecutive failures needed to turn critical
initial_status State right after registration: critical by default, so unproven instances get no traffic
deregister_critical_service_after Remove the service if its check stays critical this long (minimum one minute)

The defaults act on the first result: one failed probe makes the check critical, and one success makes it passing. The success_before_passing and failures_before_critical settings add hysteresis: a transient blip does not change the state, and a recovering service has to prove itself.

Estimating detection time

If an instance dies right after a successful probe, the worst case before Consul reports it critical is roughly:

TEXT
worst case ~ interval x failures_before_critical + timeout

With interval = 10s, failures_before_critical = 3 and a timeout of 3s, that is up to 33 seconds. DNS answers cached for a TTL add more, and clients that cache for a minute keep calling the dead instance even longer. Fast failover needs all of these to be short, or client-side retries to a second instance.

A short interval is not free: a 1-second HTTP check on 5,000 instances is 5,000 requests per second against your health endpoints, plus CPU on the agents and more catalog writes whenever states change. Typical values are 5 to 30 seconds for the interval.

flowchart LR
  P["Probe result"] --> C{"Consecutive failures >= failures_before_critical?"}
  C -->|no| Keep["Stay passing"]
  C -->|yes| Crit["critical: hidden from DNS and ?passing"]
  Crit --> S{"Consecutive successes >= success_before_passing?"}
  S -->|yes| Pass["passing again"]
  S -->|no| Crit

Counters on both sides stop the state from jumping on single blips.

Why instances start critical

By default a new check is critical until its first success. That prevents a starting application from receiving traffic before it is ready, but it can also delay the first appearance for one interval. If you control the application's readiness yourself, set initial_status = "passing"; if the app needs warm-up, keep the default and make the health endpoint return a non-2xx status until it is ready.

TTL checks and heartbeats

A TTL check has a time-to-live and no interval. The application is responsible for refreshing it:

HCL
check {
  name = "worker heartbeat"
  ttl  = "30s"
}
Terminal
# healthy heartbeat, resets the countdown
$ curl --request PUT http://127.0.0.1:8500/v1/agent/check/pass/worker-heartbeat
# report status and details in one call
$ curl --request PUT http://127.0.0.1:8500/v1/agent/check/update/worker-heartbeat \
    --data '{"Status": "passing", "Output": "processed 1204 jobs"}'

Rules of thumb for TTLs:

  • Send the heartbeat at a third to a half of the TTL, so one missed or delayed call does not turn the check critical.
  • If a heartbeat does not arrive in time, the check becomes critical automatically.
  • Use warn for degraded states and fail when the worker knows it cannot do its job, rather than waiting for the TTL to expire.

Cleaning up dead services

A crashed container that never deregistered would stay in the catalog, critical, forever. The deregister_critical_service_after setting makes the agent remove the service after it has been critical for the given time:

HCL
deregister_critical_service_after = "10m"

Choose a value longer than normal restarts or deployments, or services will vanish during a slow rollout and need re-registering.

Stand-in 1: the state machine

A simulation of a check with failures_before_critical = 3 and success_before_passing = 2, driven by a sequence of probe results (1 success, 0 failure). Compare it with the naive settings (1 and 1) on the same noisy input.

Python Stand-in: check state machine

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in 2: detection time and TTL heartbeats

First, the detection-time formula as a calculator for a few settings. Then a TTL timeline: a worker heartbeats every 10 seconds against a 30-second TTL and stops at t=65, so the check turns critical when the TTL runs out.

Python Stand-in: detection time and TTL

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A very short TTL combined with a busy application can create false failures: if the process is paused for longer than the TTL, the check turns critical even though nothing is wrong. Send heartbeats from a dedicated lightweight thread or timer, and choose a TTL that tolerates a few missed beats.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. interval=10s, failures_before_critical=3 and timeout=2s. What is the worst case time to detect a dead instance?

    10 x 3 + 2 = 32 seconds, not counting DNS or client caching.
  2. 2. What problem does success_before_passing help prevent?

    Requiring several consecutive successes stops the state from bouncing after one lucky probe.
  3. 3. How often should a worker heartbeat a TTL check with ttl=30s?

    Heartbeating at a third to half of the TTL leaves room for delays.
  4. 4. What does deregister_critical_service_after do?

    It cleans up services that never deregistered, such as crashed containers.

Summary

  • interval and timeout set probe frequency; shorter means faster detection but more load.
  • failures_before_critical and success_before_passing add hysteresis against flapping.
  • Worst-case detection is roughly interval x failures + timeout, plus DNS and client caching.
  • TTL checks need heartbeats at a fraction of the TTL; use warn and fail for explicit states.
  • Use deregister_critical_service_after (at least one minute) to clean up services that never deregistered.

Next: Health States, Node Checks and Maintenance Mode