Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 78 ยท Services and Networking

DNS and Service Discovery

How Kubernetes DNS works: CoreDNS, Service and Pod names, search domains and ndots, SRV records, and a practical playbook for debugging DNS failures.

In a cluster, applications almost never use IP addresses. A web Pod connects to api or postgres.shop, and something translates that name into an address. That something is the cluster's DNS service, and "service discovery" in Kubernetes mostly means "DNS plus Services". It is simple when it works and baffling when it does not, so it pays to understand exactly how a short name like api becomes a real lookup.

In this lesson you will meet CoreDNS, learn the names that exist for Services and Pods, decode the search and ndots settings that every Pod gets, and see why an innocent lookup of example.com can cost several DNS queries. A Python resolver stand-in lets you watch the expansion step by step.

The commands and files below are read-only; this site has no cluster. The Python resolver is a stand-in that follows the same expansion rules as the C library inside a container, but it queries a dictionary instead of a network.

Who answers DNS?

CoreDNS Runs as Pods in kube-system.
kube-dns Service One stable address for the DNS Pods.
resolv.conf Every Pod is pointed at that address.
1 / 2

CoreDNS and the kube-dns Service

Kubernetes DNS is provided by CoreDNS, a small DNS server run as an ordinary Deployment in the kube-system namespace. It watches the Kubernetes API, so whenever a Service or endpoint changes, its answers change too. It is exposed through a Service that is historically called kube-dns (the name stuck even though the software changed), at a fixed address such as 10.96.0.10.

The kubelet configures every Pod so that its DNS queries go there. You can see the result inside any container:

Terminal
kubectl exec -it web-6d5f9-abcde -n shop -- cat /etc/resolv.conf
TEXT
search shop.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

This is controlled by the Pod field dnsPolicy, whose default ClusterFirst means "use cluster DNS for cluster names and forward everything else to the node's upstream resolvers". Setting dnsPolicy: Default (confusingly, not the default) inherits the node's resolver configuration instead, and None lets you supply everything through dnsConfig.

CoreDNS itself is configured by a Corefile, kept in a ConfigMap. The line to know is the forward . /etc/resolv.conf plugin, which sends non-cluster names upstream, and the kubernetes cluster.local plugin, which answers cluster names.

The names that exist

Services get an A (and AAAA) record:

TEXT
<service>.<namespace>.svc.cluster.local     ->  the ClusterIP

A headless Service (see NodePort, LoadBalancer and ExternalName) returns the IPs of its ready Pods instead of one virtual IP. Pods of a StatefulSet behind a headless Service also get individual, stable names:

TEXT
db-0.db.shop.svc.cluster.local   ->  the IP of Pod db-0

Named ports produce SRV records, which carry a port number along with the host name:

TEXT
_postgres._tcp.db.shop.svc.cluster.local

SRV records are rarely used by application code but are useful for clients that discover both address and port from DNS.

Pods also get records based on their IP address with the dots replaced by dashes, such as 10-244-1-5.shop.pod.cluster.local. You will almost never use these; they exist mainly for compatibility.

The suffix cluster.local is the default cluster domain. Administrators may change it, so avoid hard-coding the whole suffix in application code; short names are more portable.

Search domains and ndots

Why does api work at all when no record is called api? Because of the search line in resolv.conf. When you look up a name, the resolver library treats it in one of two ways, depending on ndots:

  • If the name has fewer than ndots dots, the resolver first tries the name with each search domain appended, in order, and tries the name as written last.
  • If it has ndots or more dots, it tries the name as written first.

Kubernetes sets ndots:5. So a Pod in namespace shop looking up api tries api.shop.svc.cluster.local first, which exists, and is done. A lookup of gateway.payments (one dot) tries gateway.payments.shop.svc.cluster.local (does not exist), then gateway.payments.svc.cluster.local (exists).

The same rule has a cost. Looking up an external name such as example.com (one dot, fewer than five) causes the resolver to try:

  1. example.com.shop.svc.cluster.local (fails)
  2. example.com.svc.cluster.local (fails)
  3. example.com.cluster.local (fails)
  4. example.com (succeeds)

Each failed lookup is a wasted query, and resolvers often ask for both A and AAAA records, doubling the traffic. Busy applications that call many external hosts can generate a surprising amount of DNS load and latency this way. Remedies:

  • Use fully qualified names with a trailing dot (example.com.) in configuration for external hosts. A trailing dot marks the name as absolute and skips the search list.
  • Lower ndots for the Pod through dnsConfig:
YAML
spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"
  • Cache closer to the Pod with NodeLocal DNSCache, a per-node cache, and keep CoreDNS replicas scaled for the load.

Be careful when lowering ndots: with a low value, a name like gateway.payments would be tried as a real external domain name first.

Stand-in: a resolver that expands short names

The stand-in below implements the rule above and a tiny "zone" of three records, so you can see every query. It prints each attempt, and how many queries a lookup needed.

Python Expand names through the search list

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The comparison is the point: the plain lookup of example.com costs four queries, while the absolute form or a lower ndots costs one.

Stand-in: building the names

This second stand-in produces the Service, StatefulSet Pod, Pod and SRV names from their parts, which is a handy way to check a name you are about to put into a configuration file.

Python Compose cluster DNS names

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Debugging DNS

When a name does not resolve, work from the inside out:

  1. Test from a Pod, not from your laptop. A throwaway Pod with DNS tools is easiest:
Terminal
kubectl run dnsutils --rm -it --restart=Never \
  --image=registry.k8s.io/e2e-test-images/agnhost:2.39 -- nslookup api.shop
  1. Check the Service exists and has endpoints (kubectl get svc,endpointslices -n shop). A name that resolves to an address but refuses connections is a Service problem, not a DNS problem.
  2. Check resolv.conf in the failing Pod (cat /etc/resolv.conf): is the nameserver the cluster DNS address and is the search list what you expect? A Pod with dnsPolicy: Default has none of the cluster names.
  3. Check CoreDNS is healthy: kubectl get pods -n kube-system -l k8s-app=kube-dns and kubectl logs -n kube-system -l k8s-app=kube-dns.
  4. Check the namespace. A short name only searches the Pod's own namespace, so cross-namespace calls need name.namespace.
  5. Intermittent failures and slowness usually point at overloaded CoreDNS, ndots multiplication, or blocked DNS traffic: a NetworkPolicy that does not allow UDP and TCP port 53 to kube-dns breaks every name lookup (see Network Policies).

Common mistakes

  • Testing a cluster name from outside the cluster and concluding that DNS is broken.
  • Forgetting the namespace when calling a Service in another namespace.
  • Hard-coding cluster.local in applications, which breaks on clusters with another domain.
  • Ignoring ndots and then wondering why external calls are slow under load.
  • Blocking port 53 in a default-deny NetworkPolicy.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A Pod in namespace shop looks up the name api. Which name is tried first with the default settings?

    api has fewer than five dots, so the first search domain, shop.svc.cluster.local, is appended and tried before the bare name.
  2. 2. How many queries does an uncached lookup of example.com need in a Pod with three search domains and ndots:5, assuming the first three search-list names fail?

    Three failing search-list attempts plus the final bare name make four queries (more if both A and AAAA are requested).
  3. 3. How do you make the resolver skip the search list for an external host?

    A trailing dot marks the name as fully qualified (absolute), so no search domains are appended.
  4. 4. What does a headless Service return when you look up its name?

    Without a cluster IP, DNS returns one record per ready Pod, so clients can see and choose individual instances.

Summary

  • CoreDNS answers cluster DNS; Pods reach it through the kube-dns Service via /etc/resolv.conf.
  • Services are named <service>.<namespace>.svc.cluster.local; headless Services and StatefulSet Pods have per-Pod records; named ports have SRV records.
  • The search list and ndots:5 make short names work, and make external lookups cost extra queries.
  • Use trailing dots, a lower ndots or NodeLocal DNSCache to cut DNS overhead.
  • Debug DNS from inside a Pod, then check Service endpoints, resolv.conf, CoreDNS health and NetworkPolicies.

Next: what happens after DNS hands back a ClusterIP, in How Services Really Work: kube-proxy.