Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 37 ยท ACLs and Security

Consul Gossip Encryption and TLS

Secure a Consul cluster: generate a gossip key with consul keygen, rotate it with the keyring, create a CA and certificates with consul tls, and configure verification.

ACLs control who may do what. Encryption controls who can listen. A Consul cluster has two kinds of traffic to protect: the gossip messages that agents exchange over UDP and TCP to track membership, and the RPC and HTTP traffic that carries real data. Gossip uses a shared symmetric key; RPC, HTTP and gRPC use TLS certificates.

The product cannot run on this site. Commands and configuration are read-only, and the runnable examples are stand-ins for the concepts.

Ports to know

Port Protocol Used for
8300 TCP Server RPC (agents to servers, server to server)
8301 TCP and UDP LAN gossip (Serf)
8302 TCP and UDP WAN gossip between datacenters
8500 TCP HTTP API (plaintext)
8501 TCP HTTPS API
8502 / 8503 TCP gRPC / gRPC with TLS (used by Envoy)
8600 TCP and UDP DNS
21000-21255 TCP Default sidecar proxy ports

Firewalls must allow the cluster-internal ports between agents. Expose 8501 and 8503 rather than 8500 and 8502 once TLS is on.

Gossip encryption

Gossip messages are encrypted and authenticated with a 32-byte symmetric key, shared by every agent in a datacenter. Generate one:

Terminal
consul keygen

It prints a base64 string. Put it in every agent's configuration:

HCL
encrypt = "pUqJrVyVRj5jsiYEkM/tFQYfWyJIv4s3XkvDwy7Cu5s="

(That sample is not a secret; generate your own.) For a new cluster, set it before the first start. For a running cluster, use the keyring, which rotates without downtime:

Terminal
consul keyring -list
consul keyring -install "<new key>"
consul keyring -use "<new key>"
consul keyring -remove "<old key>"

The sequence matters:

  1. Install the new key on all agents. Agents can now decrypt it but still send with the old one.
  2. Use it as the primary key. All agents now encrypt with the new key and can still read the old.
  3. Remove the old key once -list shows every agent has the new one as primary.

Skipping step 1 makes some agents unable to read others' messages and can partition the cluster.

Python Stand-in: keyring rotation order

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

TLS: certificate authority and certificates

TLS needs a certificate authority (CA). Consul's CLI can create one for you, which is fine for small clusters and labs; use your organization's PKI or Vault for production.

Terminal
consul tls ca create
consul tls cert create -server -dc dc1
consul tls cert create -client -dc dc1

This creates files such as consul-agent-ca.pem (the CA certificate), consul-agent-ca-key.pem (the CA private key, guard it), dc1-server-consul-0.pem and its key. The server certificate carries the name server.dc1.consul (domain consul, datacenter dc1), which is what clients verify.

Configure agents with the tls block:

HCL
tls {
  defaults {
    ca_file         = "/etc/consul.d/certs/consul-agent-ca.pem"
    cert_file       = "/etc/consul.d/certs/dc1-server-consul-0.pem"
    key_file        = "/etc/consul.d/certs/dc1-server-consul-0-key.pem"
    verify_incoming = true
    verify_outgoing = true
  }
  internal_rpc {
    verify_server_hostname = true
  }
}

ports {
  http  = -1
  https = 8501
}

What the options do:

  • verify_outgoing: this agent presents its certificate and checks the other side's.
  • verify_incoming: this agent requires valid client certificates on incoming connections. You usually disable it for the HTTPS API on clients that serve browsers, with https { verify_incoming = false } and rely on ACL tokens instead.
  • verify_server_hostname: a server certificate must be named server.<dc>.consul, which stops a compromised client from posing as a server.
  • ports.http = -1: disables the plaintext API once you use HTTPS.

Auto-encrypt and auto-config

Distributing certificates to every client is a chore. Auto-encrypt lets servers hand client agents their TLS certificates at startup:

HCL
# servers
auto_encrypt { allow_tls = true }

# clients
auto_encrypt { tls = true }

Clients then need only the CA certificate (and a token to join when ACLs are on). Auto-config goes further and also delivers the gossip key and an ACL token, using a JWT the client presents. Both reduce secret sprawl in configuration management.

Mesh certificates are separate

The leaf certificates that identify services in the mesh (see Service Mesh Concepts) come from the mesh CA, the built-in one by default or Vault, and are unrelated to the agent TLS certificates above. Rotate them independently.

Python Stand-in: certificate expiry planner

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Track certificate expiry with monitoring; an expired server certificate is one of the classic outages.

Rolling out safely

  1. Enable gossip encryption in permissive mode: set encrypt with encrypt_verify_incoming = false and encrypt_verify_outgoing = false, restart agents, then tighten (both true) once all have the key.
  2. Enable TLS with verify_outgoing = false / verify_incoming = false first, then turn verification on.
  3. Turn on ACLs after the transport is secure so tokens never travel in plaintext.
  4. Keep backups of the CA key offline.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does consul keygen produce?

    The key goes into the encrypt setting (or the keyring) on every agent.
  2. 2. What is the correct order to rotate a gossip key?

    Every agent must hold the new key before it becomes primary, and the old key goes last.
  3. 3. What does verify_server_hostname = true enforce?

    It prevents a client certificate from being accepted as a server.
  4. 4. Which port serves the HTTPS API by convention?

    8501 is the HTTPS port; 8500 is plaintext HTTP.
  5. 5. What does auto-encrypt do?

    Clients need only the CA certificate; servers provide the client certificate.

Summary

  • Gossip is protected by one shared key (consul keygen, encrypt) rotated with consul keyring in install, use, remove order.
  • TLS needs a CA and server/client certificates (consul tls ca create, consul tls cert create) and verify_incoming, verify_outgoing, verify_server_hostname.
  • Auto-encrypt and auto-config reduce certificate and secret distribution work.
  • Roll out in stages and monitor certificate expiry; mesh leaf certificates are separate.

Next: WAN Federation and Datacenters