Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 37 ยท Key-Value and Config

Watches and Blocking Queries

React to Consul changes without polling: blocking queries with index and wait, X-Consul-Index rules, the consul watch command, watch types, handlers and event firing.

Configuration changes and service changes should reach your application quickly, but asking "anything new?" every second wastes requests and still reacts late. Consul solves this with blocking queries: the client asks "tell me when this data is different from what I have", and the server holds the request open until something changes or a timeout passes. A change is seen almost immediately and an idle system costs one open request per watcher.

The requests and commands are read-only here; the stand-ins simulate a blocking server and the client loop rules.

How a blocking query works

Nearly every read endpoint (KV, catalog, health, sessions, config entries) returns an X-Consul-Index header. Pass that number back as ?index= on the next call:

Terminal
# first call: no index, returns immediately, note the X-Consul-Index header
$ curl -i http://127.0.0.1:8500/v1/kv/app/config
# X-Consul-Index: 4801

# second call: blocks until the data changes or 30 seconds pass
$ curl "http://127.0.0.1:8500/v1/kv/app/config?index=4801&wait=30s"

When the data changes, the call returns right away with the new data and a new, higher X-Consul-Index. If nothing changes, it returns when wait expires with the same data and the same index. The default wait is five minutes, the maximum ten; the server adds a little random jitter so many watchers do not all reconnect at the same instant.

sequenceDiagram
  participant C as Client
  participant S as Consul server
  C->>S: GET /kv/app/config
  S-->>C: value v1, X-Consul-Index 4801
  C->>S: GET ...?index=4801&wait=30s
  Note over S: request is held open
  S-->>C: value v2, X-Consul-Index 4825 (data changed)
  C->>S: GET ...?index=4825&wait=30s
  Note over S: waits again

One long-lived request replaces constant polling.

The rules of the index

A correct client follows these rules from Consul's design:

  1. Treat the index as opaque. Do not do arithmetic on it or assume it is the same across endpoints; each query path has its own sequence.
  2. Same index returned = nothing changed. A timeout returns the same index; do not treat that reply as an update.
  3. Index went backwards = reset. If the new index is lower than the previous one (it can after a snapshot restore or leader changes), start again with index=0 (a normal, non-blocking read).
  4. Never use an index below 1 in a blocking request; an index of 0 or less means a non-blocking read.
  5. Add your own delay on errors. If a request fails, back off (with jitter) before retrying, so a broken server is not hit by a thundering herd.
  6. Use a request timeout slightly larger than wait to avoid cutting off a legitimate long poll.

Note that blocking queries return when any data under the queried path changes (for a prefix, any key in it), and the index may increase for reasons that do not alter your data. Compare content if you only care about real changes.

Stale and blocking together

Blocking queries are served by the leader by default. Add stale to let followers hold them, which spreads load across servers with the small risk of a slightly late update. Large fleets of watchers should use stale (and agent caching, where applicable) to avoid overloading the leader.

consul watch

You do not have to write the loop yourself. The consul watch command runs one for you and calls a handler whenever the data changes:

Terminal
$ consul watch -type=key -key=app/config ./reload.sh
$ consul watch -type=keyprefix -prefix=app/ ./sync-all.sh
$ consul watch -type=service -service=web -passing-only ./rebuild-proxy.sh
$ consul watch -type=services ./inventory.sh
$ consul watch -type=nodes ./nodes.sh
$ consul watch -type=checks -state=critical ./alert.sh
$ consul watch -type=event -name=deploy ./on-deploy.sh
Type Fires when
key One key changes
keyprefix Any key under a prefix changes
services The list of services changes
service Instances or health of one service change
nodes The node list changes
checks Check states change
event A user event arrives

The handler receives the new data as JSON on standard input, so it can be any program. The handler runs on each change, so it must be quick and idempotent. Watches can also be defined permanently in an agent's configuration file:

HCL
watches = [
  {
    type    = "key"
    key     = "app/config"
    handler = "/usr/local/bin/reload-app.sh"
  }
]

User events

consul event broadcasts a small message (under 512 bytes) to the agents, a lightweight way to say "deploy now" or "refresh caches":

Terminal
$ consul event -name=deploy "release-2.4.1"

Events are delivered with best-effort semantics through gossip; do not rely on them for anything that must not be lost. Use the KV store or an external queue for that.

Stand-in 1: a blocking server and client

A model of a server that holds requests until the index moves past the client's, with logical time standing in for seconds. It demonstrates immediate wake-up and the timeout path.

Python Stand-in: blocking query

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in 2: the client-side index rules

A helper that applies rules 2 and 3 to a stream of replies. Feed it the indexes a client sees and it decides what to do next.

Python Stand-in: index handling

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Instead of coding loops or running consul watch by hand, tools like Consul Template and the built-in agent cache for service lookups do the blocking and index logic for you. The next lesson uses them for configuration.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does ?index=4801&wait=30s mean?

    The server holds the request until the data's index is greater than 4801 or the wait expires.
  2. 2. A blocking query returns the same X-Consul-Index you sent. What happened?

    An unchanged index means the wait expired without a change.
  3. 3. The returned index is lower than the one you previously saw. What should you do?

    Indexes can move backwards after restores or leader changes; restarting from 0 is the safe handling.
  4. 4. How does a consul watch handler receive the changed data?

    The new data is sent to the handler's standard input as JSON.

Summary

  • Blocking queries (index + wait) turn polling into near-instant change notification.
  • Use the X-Consul-Index header: same = no change, lower = reset to 0, higher = process.
  • Back off on errors, use request timeouts larger than wait, and use stale to spread load.
  • consul watch runs the loop and invokes a handler with JSON on stdin; watches can live in agent config.
  • User events are best-effort broadcasts, not durable messages.

Next: Configuration Patterns and Consul Template