Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français

CoreDNS, the internal DNS of Kubernetes

By updated

Improve this page

CoreDNS is the internal DNS server of Kubernetes: it turns the name of a Service, such as web.ws-dns.svc.cluster.local, into an IP address, so that Pods reach each other by a stable name rather than by an address that changes. This lesson follows a name resolution from a Pod to CoreDNS, reads and changes its configuration, then breaks DNS on purpose to teach you how to diagnose it. It is for readers who can already reach a Service by its name (see Services) and want to know what to look at when resolution fails.

What you will learn

  • Describe the path of a name resolution from a Pod to CoreDNS
  • Write the full DNS name of a Service, of a Pod and of a Pod behind a headless Service
  • Check that CoreDNS works: Deployment, Pods and the kube-dns Service
  • Read the Corefile and state the role of its main plugins
  • Add static entries or an external DNS, and restore the original configuration
  • Diagnose a resolution failure from a debug Pod
  • Read the Prometheus metrics of CoreDNS
  • Choose a dnsPolicy and write a dnsConfig for a Pod

Prerequisites

You have read Pod Networking and you have the four-node demo-reseau cluster created in Pod Networking (kind 0.33.0, nodes in v1.37.0, CoreDNS 1.14.6). Save the following file as dns-base.yaml: it describes a web Deployment and its Service in the ws-dns namespace, plus a client Pod named client in each of the ws-dns and ws-dns-b namespaces. The commands that follow create the namespaces, then apply the file.

apiVersion: apps/v1
kind: Deployment
metadata:
name: web
namespace: ws-dns
spec:
replicas: 2
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: nginx
image: nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: web
namespace: ws-dns
spec:
selector:
app: web
ports:
- port: 80
targetPort: 80
---
apiVersion: v1
kind: Pod
metadata:
name: client
namespace: ws-dns
spec:
containers:
- name: busybox
image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4
command: ["sleep", "3600"]
---
apiVersion: v1
kind: Pod
metadata:
name: client
namespace: ws-dns-b
spec:
containers:
- name: busybox
image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4
command: ["sleep", "3600"]
Terminal window
kubectl create namespace ws-dns
kubectl create namespace ws-dns-b
kubectl apply -f dns-base.yaml
kubectl -n ws-dns rollout status deployment/web --timeout=90s

Some names in this lesson are in French on purpose, so that the commands and outputs stay those of the tested cluster: demo-reseau means "network-demo", nexiste-pas "does-not-exist", vieux.corp.test "old.corp.test", coredns.sauve.yaml "coredns.saved.yaml", Corefile.faute "Corefile.mistake", Corefile.cas1 "Corefile.case1", heritage "legacy", web-ancien "web-old" and exemple.org "example.org".

How does a Pod resolve a name in Kubernetes?

Kubernetes hands out addresses that change: a recreated Pod gets a new address, and so does a Service that is deleted and created again. The internal DNS maps a stable name to the address of the moment. CoreDNS stores nothing: it reads the Services and the EndpointSlices from the Kubernetes API and turns them into DNS answers.

DNS architecture

Three elements take part, and mixing them up makes troubleshooting harder: the /etc/resolv.conf file that the kubelet writes in every Pod, the kube-dns Service that holds a fixed address, and the CoreDNS Pods that actually answer. The kube-dns name of the Service is a legacy: it does designate CoreDNS. Notice on the diagram that Pods only know the address of the Service.

Three Pods whose /etc/resolv.conf file points to the DNS IP send their queries to the kube-dns Service, which holds the ClusterIP 10.96.0.10:53 and forwards them to the CoreDNS Pods managed by a Deployment.Cluster KubernetesDNS queriesPod Aresolv.conf: DNS IPService kube-dnsClusterIP 10.96.0.10:53Pod Bresolv.conf: DNS IPPod Cresolv.conf: DNS IPPods CoreDNS(Deployment)
Source · SVG
Esc to close, + and − to zoom, scroll to move around. 100 %

Resolution flow

Following a query from end to end gives you the map of the failure points, because each step can fail on its own. When a Pod wants to reach web.ws-dns.svc.cluster.local:

  1. The Pod reads /etc/resolv.conf, which points to the address of the kube-dns Service.
  2. The query goes to that Service, which forwards it to one of the CoreDNS Pods.
  3. The kubernetes plugin of CoreDNS answers from the objects of the cluster.
  4. The Pod receives the ClusterIP of the Service and contacts it.

Check that CoreDNS works

Before suspecting a configuration, check that the chain exists: the Deployment, its Pods, the Service and its endpoints. A few commands that take a few seconds rule out the most obvious causes.

Deployment status

The coredns Deployment declares two replicas on a kubeadm or kind cluster. The AVAILABLE column must equal READY.

Terminal window
kubectl -n kube-system get deployment coredns
kubectl -n kube-system get deployment coredns -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
NAME READY UP-TO-DATE AVAILABLE AGE
coredns 2/2 2 2 8m57s
registry.k8s.io/coredns/coredns:v1.14.6

2/2 means both replicas answer, and the second command gives the CoreDNS version, here v1.14.6.

Pod status

The selector label is k8s-app=kube-dns while the Pods are named coredns: the log and diagnostic commands expect the label, not the name.

Terminal window
kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-559f6c778d-gckk9 1/1 Running 0 8m51s 10.244.0.2 demo-reseau-control-plane <none> <none>
coredns-559f6c778d-gcx8k 1/1 Running 0 8m51s 10.244.0.3 demo-reseau-control-plane <none> <none>

Look at READY, STATUS and RESTARTS. Two Pods that are 1/1 Running with no restart mean healthy DNS. Here, both run on the control plane.

The kube-dns Service

The kubelet writes the address of this Service in the /etc/resolv.conf of every Pod. Port 9153 serves the metrics, and the two port 53 entries expose DNS over UDP and over TCP.

Terminal window
kubectl -n kube-system get svc kube-dns
kubectl -n kube-system get endpointslices -l kubernetes.io/service-name=kube-dns
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 8m57s
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
kube-dns-bhzgf IPv4 53,53,9153 10.244.0.3,10.244.0.2 8m51s

The address 10.96.0.10 is the one of this cluster: it depends on the Service range, so check it with the first command. The ENDPOINTS column of the EndpointSlice lists the addresses of the two CoreDNS Pods. A Service without endpoints answers nobody, and you will see below what that looks like.

Which DNS name for a Service, a Pod or a headless Pod?

Kubernetes follows a naming convention that is the same on every cluster. The cluster domain is cluster.local by default. Three families of objects each have their own form: Services, Pods and Pods behind a headless Service.

Services

The full name of a Service is <service>.<namespace>.svc.cluster.local. The short forms work thanks to the search line of the Pod's /etc/resolv.conf, which completes incomplete names: they therefore resolve only from a Pod, never from a node.

FormSpellingExample
Full name<service>.<namespace>.svc.cluster.localweb.ws-dns.svc.cluster.local
Same namespace<service>web
Other namespace<service>.<namespace>web.ws-dns

From the client Pod of the ws-dns namespace, first read the resolver file, then resolve the full name:

Terminal window
kubectl -n ws-dns exec client -- cat /etc/resolv.conf
kubectl -n ws-dns exec client -- nslookup web.ws-dns.svc.cluster.local; echo "rc=$?"
search ws-dns.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5
Server: 10.96.0.10
Address: 10.96.0.10:53
Name: web.ws-dns.svc.cluster.local
Address: 10.96.54.40
rc=0

Look at the nameserver: it is the ClusterIP of the kube-dns Service. The search line puts the namespace of the Pod first, which explains why web is enough in ws-dns and not elsewhere. With the full name, the answer comes back cleanly and rc=0.

With the short name web, the result depends on the tool. wget http://web reaches nginx, but the busybox nslookup finds the address while surrounding it with NXDOMAIN lines for the suffixes that fail, then exits with rc=1: draw no conclusion about DNS from it, the reason is detailed in Pod Networking. From the other namespace, the short name no longer resolves, while the full name gives the same address:

Terminal window
kubectl -n ws-dns-b exec client -- wget -qO- -T 3 http://web; echo "rc=$?"
kubectl -n ws-dns-b exec client -- nslookup web.ws-dns.svc.cluster.local; echo "rc=$?"
wget: bad address 'web'
command terminated with exit code 1
rc=1
Server: 10.96.0.10
Address: 10.96.0.10:53
Name: web.ws-dns.svc.cluster.local
Address: 10.96.54.40
rc=0

bad address 'web' means that web does not exist in any namespace of the Pod's search list, here ws-dns-b. The full name depends on no list: when diagnosing, always write it in full.

Pods

Kubernetes can publish a name for each Pod, provided the kubernetes plugin of CoreDNS has the pods insecure option, which is the case here (see the Corefile below). The form is <address-with-dashes>.<namespace>.pod.cluster.local. Resolve the name of an address that no Pod holds, then the same name with a namespace that does not exist:

Terminal window
kubectl -n ws-dns exec client -- nslookup 10-244-99-99.ws-dns.pod.cluster.local; echo "rc=$?"
kubectl -n ws-dns exec client -- nslookup 10-244-99-99.nexiste-pas.pod.cluster.local; echo "rc=$?"
Server: 10.96.0.10
Address: 10.96.0.10:53
Name: 10-244-99-99.ws-dns.pod.cluster.local
Address: 10.244.99.99
rc=0
Server: 10.96.0.10
Address: 10.96.0.10:53
** server can't find 10-244-99-99.nexiste-pas.pod.cluster.local: NXDOMAIN
** server can't find 10-244-99-99.nexiste-pas.pod.cluster.local: NXDOMAIN
command terminated with exit code 1
rc=1

No Pod holds 10.244.99.99, and yet CoreDNS answers 10.244.99.99. With pods insecure, CoreDNS does not ask Kubernetes about a Pod name: it copies the address contained in the name. A positive answer therefore does not prove that the Pod exists: it is a poor presence test. The namespace, on the other hand, is checked: a namespace that does not exist gives NXDOMAIN.

Headless Services

A headless Service, declared with clusterIP: None, gets no virtual address. CoreDNS then returns the addresses of the Pods directly, and also gives each of them a stable name, which lets a client target a specific instance, as with a replicated database. Save this file as headless.yaml: a StatefulSet of two db Pods and the headless db Service (see StatefulSets).

apiVersion: v1
kind: Service
metadata:
name: db
namespace: ws-dns
spec:
clusterIP: None
selector:
app: db
ports:
- port: 80
name: http
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: db
namespace: ws-dns
spec:
serviceName: db
replicas: 2
selector:
matchLabels:
app: db
template:
metadata:
labels:
app: db
spec:
containers:
- name: nginx
image: nginx:1.30.5@sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb
ports:
- containerPort: 80
Terminal window
kubectl apply -f headless.yaml
kubectl -n ws-dns rollout status statefulset/db --timeout=120s
kubectl -n ws-dns exec client -- nslookup db.ws-dns.svc.cluster.local
kubectl -n ws-dns exec client -- nslookup db-0.db.ws-dns.svc.cluster.local
Name: db.ws-dns.svc.cluster.local
Address: 10.244.2.3
Name: db.ws-dns.svc.cluster.local
Address: 10.244.1.6
Name: db-0.db.ws-dns.svc.cluster.local
Address: 10.244.1.6

The Server lines are omitted. The Service name returns two addresses, those of the Pods, and no ClusterIP. The <pod>.<service>.<namespace>.svc.cluster.local name returns a single address, that of db-0, the same as kubectl -n ws-dns get pod db-0 -o jsonpath='{.status.podIP}'.

What does the Corefile contain?

CoreDNS is configured by a ConfigMap named coredns in the kube-system namespace, which holds a single file, the Corefile. It describes a chain of plugins. Changing this ConfigMap changes the behavior of the DNS of the whole cluster, without redeploying anything.

Show the current configuration

Always read the real configuration rather than assuming the one from your distribution: each one may ship a variant.

Terminal window
kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}'

The output is the Corefile below.

A common Corefile example (kubeadm)

Here is the Corefile set up by kind, which installs CoreDNS with kubeadm, here with CoreDNS 1.14.6:

.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf {
max_concurrent 1000
}
cache 30 {
disable success cluster.local
disable denial cluster.local
}
loop
reload
loadbalance
}

Look at two blocks. In kubernetes, the pods insecure option makes Pod names resolvable, as seen above. In cache 30, the two disable lines exclude cluster.local from the cache: according to the documentation of the cache plugin, CoreDNS then does not keep internal answers in memory.

If a Pod still reaches the old address of a Service whose ClusterIP has changed, the CoreDNS cache is therefore not the cause: look for a cache on the application side or in the system. The 30-second cache only applies to external names.

What the plugins do

Two plugins do most of the work: kubernetes answers for internal names, and forward sends everything else to the configured resolvers, here the node's /etc/resolv.conf file. The others bring observability, caching and robustness. The roles below are those of the CoreDNS plugin documentation.

PluginRole
errorswrites errors to the log
health and readyexpose the health and readiness of the Pod
kubernetesanswers for the names of the cluster
prometheusexposes the metrics on port 9153
forwardforwards the other queries to the upstream resolvers
cachecaches the external answers, 30 seconds
loopdetects forwarding loops
reloadreloads the Corefile when the ConfigMap changes
loadbalancerotates the order of the addresses in an answer

The reload plugin does not act right away: on a kind cluster, a valid change is picked up after 75 to 89 seconds (next section).

How do you change CoreDNS without breaking the cluster's DNS?

A syntax error in the Corefile can cut resolution for the whole cluster. So keep a copy of the ConfigMap before every change, with the command that follows.

Terminal window
kubectl -n kube-system get configmap coredns -o yaml > coredns.sauve.yaml

Add a custom external DNS

To resolve a company-internal domain with its own DNS, the Kubernetes documentation on customizing DNS adds a server block before the main block. The corp.test domain and the 10.0.0.53 address are examples to replace:

corp.test:53 {
errors
cache 30
forward . 10.0.0.53
}

Add static entries

For a host name that exists in no DNS, the hosts plugin provides static entries. Insert a hosts block before the kubernetes block, then replace the ConfigMap:

Terminal window
kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' | awk '/^ kubernetes cluster.local/ {print " hosts {"; print " 192.0.2.10 vieux.corp.test"; print " fallthrough"; print " }"} {print}' > Corefile.hosts
kubectl -n kube-system create configmap coredns --from-file=Corefile=Corefile.hosts --dry-run=client -o yaml | kubectl replace -f -
configmap/coredns replaced

The inserted block is:

hosts {
192.0.2.10 vieux.corp.test
fallthrough
}

According to the documentation of the hosts plugin, the fallthrough word lets the names absent from this block go on to the next plugins, including kubernetes. The change is not immediate: retry the resolution until it answers.

Terminal window
kubectl -n ws-dns exec client -- nslookup vieux.corp.test
Server: 10.96.0.10
Address: 10.96.0.10:53
Name: vieux.corp.test
Address: 192.0.2.10

On a kind cluster, the first answer arrives between 75 and 89 seconds after the replace: allow one to two minutes before concluding that it does not work. The log shows [INFO] Reloading then [INFO] Reloading complete.

What happens with a mistake in the Corefile?

Now replace forward with forwrd in the static entries file and apply it:

Terminal window
sed 's#forward \. /etc/resolv.conf#forwrd . /etc/resolv.conf#' Corefile.hosts > Corefile.faute
kubectl -n kube-system create configmap coredns --from-file=Corefile=Corefile.faute --dry-run=client -o yaml | kubectl replace -f -
kubectl -n kube-system logs -l k8s-app=kube-dns --prefix --tail=3

After about a minute, the log of one of the Pods contains:

[pod/coredns-64858cd7ff-chvx4/coredns] [INFO] Reloading
[pod/coredns-64858cd7ff-chvx4/coredns] [ERROR] Restart failed: /etc/coredns/Corefile:17 - Error during parsing: Unknown directive 'forwrd'
[pod/coredns-64858cd7ff-chvx4/coredns] [ERROR] plugin/reload: Corefile changed but reload failed: starting with listener file descriptors: /etc/coredns/Corefile:17 - Error during parsing: Unknown directive 'forwrd'

The reload is refused, but the Pod keeps running with the old configuration: the Pods stay Running and vieux.corp.test still resolves. A mistake therefore stays silent until the day a Pod restarts.

A Pod that starts with the mistake has no old configuration to fall back on: it loops in CrashLoopBackOff. To see it, scale CoreDNS to zero replicas with the faulty ConfigMap in place, then start two again:

Terminal window
kubectl -n kube-system scale deployment coredns --replicas=0

The error message quotes the line of forward in your file: 17 with the file of this page. The output below comes from a shorter Corefile, where forward is on line 13.

Terminal window
kubectl -n kube-system scale deployment coredns --replicas=2
sleep 30
kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=20

The output below is an excerpt: the maxprocs lines of the log are omitted.

deployment.apps/coredns scaled
NAME READY STATUS RESTARTS AGE
coredns-559f6c778d-8t7gr 0/1 CrashLoopBackOff 2 (18s ago) 30s
coredns-559f6c778d-cwdh8 0/1 CrashLoopBackOff 2 (17s ago) 30s
/etc/coredns/Corefile:13 - Error during parsing: Unknown directive 'forwrd'

This output shows what happens to new CoreDNS Pods when the ConfigMap contains a mistake: they go into CrashLoopBackOff and are never ready.

The Error during parsing: Unknown directive 'forwrd' log gives the faulty line. The kubectl -n kube-system logs -l k8s-app=kube-dns command is therefore your first reflex when the CoreDNS Pods do not start.

How do you restore the original configuration?

The obvious move, replaying the backup with replace, is refused:

Terminal window
kubectl -n kube-system replace -f coredns.sauve.yaml
Error from server (Conflict): error when replacing "coredns.sauve.yaml": Operation cannot be fulfilled on configmaps "coredns": the object has been modified; please apply your changes to the latest version and try again

The saved file contains the metadata.resourceVersion field, which dates from the backup: the object has changed since, and Kubernetes refuses to overwrite a newer version. kubectl apply -f coredns.sauve.yaml fails too, with the same Conflict. The move that works removes this line from the file before the replace:

Terminal window
sed '/resourceVersion/d' coredns.sauve.yaml | kubectl -n kube-system replace -f -
configmap/coredns replaced

If CoreDNS Pods are in CrashLoopBackOff, restart them after restoring the ConfigMap, never before: a rollout restart done with the faulty ConfigMap fixes nothing, the new Pods fail the same way.

Terminal window
kubectl -n kube-system rollout restart deployment coredns
kubectl -n kube-system rollout status deployment/coredns --timeout=120s
deployment.apps/coredns restarted
Waiting for deployment spec update to be observed...
[...]
deployment "coredns" successfully rolled out

Two new 1/1 Running Pods are ready in about 11 seconds.

Without a restart, the Pods that are already running pick up the right configuration on their own, 33 seconds after the replace on kind. This delay varies, from 33 to 89 seconds in this lesson.

How do you diagnose a DNS failure?

A DNS problem rarely shows up as such: the application reports an unknown host, a refused connection or slowness. The method is to test the resolution from a Pod, compare it with the expected behavior, then work back up the chain to CoreDNS.

Quick test with a debug Pod

The test starts from a Pod, never from a node, because only a Pod has an /etc/resolv.conf that points to CoreDNS. The client Pod of the lesson does the job.

The kubernetes.default name is a good target, since this Service exists on every cluster. With busybox, query it with a tool that goes through the C library, such as nc, because the busybox nslookup does not complete a name with a single dot:

Terminal window
kubectl -n ws-dns exec client -- nc -zv -w 3 kubernetes.default 443; echo "rc=$?"
kubernetes.default (10.96.0.1:443) open
rc=0

The open answer proves that the short name resolved to 10.96.0.1 and that the port answers. For nslookup, use the full name kubernetes.default.svc.cluster.local.

Check /etc/resolv.conf in a Pod

This file is fixed when the Pod is created: a change to the DNS configuration only reaches an existing Pod after it is recreated. Check that the nameserver matches the ClusterIP of the kube-dns Service, as seen above in the client Pod: nameserver 10.96.0.10, options ndots:5 and three cluster search suffixes, with no node suffix.

Common problems and solutions

Four situations cover most failures. Three are told apart by a single question, what resolves, and what fails? The fourth, slowness, shows no error. Follow the tree from top to bottom, with a Service name written in full: each "no" points to the section that deals with the failure.

Decision tree with three questions. If a Service name written in full does not resolve, no name resolves: check the CoreDNS Pods, the kube-dns endpoints and the resolv.conf of the Pod. If it resolves but an external name fails, check the forward line of the Corefile and the resolvers of the node. If both resolve but the connection to the address fails, DNS is not the cause: check the endpoints and the port of the Service. If everything succeeds, but slowly, look at ndots and the search list.A Pod failson a nameDoes a Service name,written in full,resolve?No name resolves:CoreDNS Pods, kube-dnsendpoints, resolv.confnoDoes an externalname resolve?yesInternal only:Corefile forward line,node resolversnoDoes the connectionto the returnedaddress succeed?yesThe name resolves, theconnection does not:Service endpoints and portnoEverything works, but slowly:ndots and the search listyes
Source · SVG
Esc to close, + and − to zoom, scroll to move around. 100 %

The Pod resolves no name

When no resolution works, neither internal nor external, the Pod does not reach CoreDNS or CoreDNS does not answer. Scale CoreDNS down to zero replicas to see the symptom. The sleep 5 gives the Pods time to stop.

Terminal window
kubectl -n kube-system scale deployment coredns --replicas=0
sleep 5
time kubectl -n ws-dns exec client -- nslookup web; echo "rc=$?"
kubectl -n kube-system get endpointslices -l kubernetes.io/service-name=kube-dns
deployment.apps/coredns scaled
nslookup: write to '10.96.0.10': Connection refused
;; connection timed out; no servers could be reached
command terminated with exit code 1
real 0m0.077s
rc=1
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
kube-dns-bhzgf IPv4 <unset> <unset> 21m

The failure is immediate: Connection refused in 0.08 seconds, not a delay to wait for. This shape keeps you from believing in a network block. The kube-dns Service now has no Pod behind it, and its EndpointSlice is empty (<unset>).

The EndpointSlice does not empty right away. On kind, it is empty five seconds after the scale, which matches the lameduck 5s of the Corefile; just after the scale, it still contains the addresses of the Pods that are stopping.

To work back from a failure, run kubectl -n kube-system get pods -l k8s-app=kube-dns, then look at the Service, then kubectl -n kube-system logs -l k8s-app=kube-dns. A NetworkPolicy that closes the egress of the Pod also blocks DNS.

Bring back two replicas with kubectl -n kube-system scale deployment coredns --replicas=2: two replicas are ready in about 5 seconds and resolution comes back.

Internal resolution works, external fails

When the names of the cluster resolve but external names do not, CoreDNS works and the failure is downstream: the forward plugin does not reach its resolvers, by default those of the node's /etc/resolv.conf file. Point forward at an address that does not answer, then compare:

Terminal window
kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' | sed 's#forward \. /etc/resolv.conf#forward . 192.0.2.1#' > Corefile.cas1
kubectl -n kube-system create configmap coredns --from-file=Corefile=Corefile.cas1 --dry-run=client -o yaml | kubectl replace -f -

After the reload (up to 89 seconds), the internal name resolves and the external name fails:

Terminal window
kubectl -n ws-dns exec client -- nslookup web.ws-dns.svc.cluster.local; echo "rc=$?"
time kubectl -n ws-dns exec client -- nslookup -timeout=3 exemple.org; echo "rc=$?"
[...]
Name: web.ws-dns.svc.cluster.local
Address: 10.96.54.40
rc=0
;; connection timed out; no servers could be reached
command terminated with exit code 1
real 0m3.189s
rc=1

The failure lasts 3.19 seconds, which is the requested timeout: it is silence, not a refusal.

The "internal OK, external KO" pair points to the forward line: check it with kubectl -n kube-system get configmap coredns -o yaml | grep forward, then the resolvers of the node, or give explicit resolvers. Then restore the backup as in the previous section.

The name resolves, but the connection fails

A name that resolves to an address does not guarantee that a Pod answers behind it. Save this heritage Service in heritage.yaml: its app: web-ancien selector matches no Pod.

apiVersion: v1
kind: Service
metadata:
name: heritage
namespace: ws-dns
spec:
selector:
app: web-ancien
ports:
- port: 80
targetPort: 80
Terminal window
kubectl apply -f heritage.yaml
kubectl -n ws-dns exec client -- nslookup heritage.ws-dns.svc.cluster.local
time kubectl -n ws-dns exec client -- wget -qO- -T 3 http://heritage; echo "rc=$?"
kubectl -n ws-dns get endpointslices -l kubernetes.io/service-name=heritage
[...]
Name: heritage.ws-dns.svc.cluster.local
Address: 10.96.90.1
wget: can't connect to remote host (10.96.90.1): Connection refused
command terminated with exit code 1
real 0m1.082s
rc=1
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
heritage-ggngl IPv4 <unset> <unset> 1s

The name resolves to 10.96.90.1, but the connection is refused (Connection refused): the EndpointSlice has no port and no address, because no Pod carries the requested label. DNS is not the cause: compare the labels of the Pods (kubectl -n ws-dns get pods --show-labels) with the selector of the Service.

Endpoints that are present are not enough either. After kubectl -n ws-dns patch service web --type merge -p '{"spec":{"ports":[{"port":80,"targetPort":8080}]}}' and five seconds of waiting, the EndpointSlice does carry port 8080 and the two addresses, but wget answers Connection refused, since nginx listens on port 80.

Right after the patch, the page still loaded: the kube-proxy rules are not updated within the second. The topic is covered in Services.

Slow external resolution: four queries instead of one

With options ndots:5, the manual page resolv.conf(5) states that a name with fewer than five dots is first tried with each suffix of the search line, and that the requested name is queried as is only last. With the three suffixes of the client Pod, an external name such as exemple.org can therefore cost four queries instead of one for each record type requested (A, AAAA), including three useless NXDOMAIN answers.

Writing the name with a trailing dot, exemple.org., declares it absolute and skips the search list. For a Pod that makes many external queries, lower ndots with a dnsConfig, as in the next section.

How do you choose dnsPolicy and dnsConfig?

Everything above concerns the server. Two fields of a Pod manifest act on the client side: dnsPolicy chooses the resolver that the Pod queries, and dnsConfig adjusts or replaces the content of its /etc/resolv.conf. They are set when the Pod is created.

dnsPolicy

The possible values are described in the Kubernetes documentation.

ValueBehavior
ClusterFirst (default)queries CoreDNS for everything
Defaulttakes over the node's resolv.conf
ClusterFirstWithHostNetfor Pods with hostNetwork: true
Noneno configuration, to be written in dnsConfig

The hostNetwork case is recognized by a precise symptom. Save this Pod as hostnet.yaml:

apiVersion: v1
kind: Pod
metadata:
name: hostnet
namespace: ws-dns
spec:
hostNetwork: true
containers:
- name: busybox
image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4
command: ["sleep", "3600"]
Terminal window
kubectl apply -f hostnet.yaml
kubectl -n ws-dns wait --for=condition=Ready pod/hostnet --timeout=60s
kubectl -n ws-dns exec hostnet -- cat /etc/resolv.conf
kubectl -n ws-dns exec hostnet -- nslookup -timeout=2 web.ws-dns.svc.cluster.local; echo "rc=$?"
nameserver 172.18.0.1
options edns0 trust-ad ndots:0
Server: 172.18.0.1
Address: 172.18.0.1:53
** server can't find web.ws-dns.svc.cluster.local: SERVFAIL
** server can't find web.ws-dns.svc.cluster.local: SERVFAIL
command terminated with exit code 1
rc=1

Look at the nameserver: 172.18.0.1 is not the ClusterIP of kube-dns, but the resolver of the kind node. The Pod inherited the node's resolver file, and the answer is SERVFAIL, not NXDOMAIN: this server does not know the cluster zone.

The code guides the diagnosis. NXDOMAIN means that a competent server answered that the name does not exist; SERVFAIL, that you are querying a server that cannot answer.

With dnsPolicy: ClusterFirstWithHostNet added under hostNetwork: true (Pod hostnet-cf), the file becomes the cluster's again (nameserver 10.96.0.10, ndots:5) and nslookup web.ws-dns.svc.cluster.local returns 10.96.54.40 with rc=0.

dnsConfig

The dnsConfig block adds to the default behavior, or replaces it entirely with dnsPolicy: "None". Save these two Pods in dnsconfig.yaml: ndots2 lowers ndots to 2, and dns-none declares a resolver and a suffix of its own.

apiVersion: v1
kind: Pod
metadata:
name: ndots2
namespace: ws-dns
spec:
dnsConfig:
options:
- name: ndots
value: "2"
containers:
- name: busybox
image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4
command: ["sleep", "3600"]
---
apiVersion: v1
kind: Pod
metadata:
name: dns-none
namespace: ws-dns
spec:
dnsPolicy: "None"
dnsConfig:
nameservers:
- 192.0.2.53
searches:
- corp.test
containers:
- name: busybox
image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4
command: ["sleep", "3600"]
Terminal window
kubectl apply -f dnsconfig.yaml
kubectl -n ws-dns wait --for=condition=Ready pod/ndots2 pod/dns-none --timeout=60s
kubectl -n ws-dns exec ndots2 -- cat /etc/resolv.conf
kubectl -n ws-dns exec dns-none -- cat /etc/resolv.conf
time kubectl -n ws-dns exec dns-none -- nslookup -timeout=2 web.ws-dns.svc.cluster.local; echo "rc=$?"
search ws-dns.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:2
search corp.test
nameserver 192.0.2.53
;; connection timed out; no servers could be reached
command terminated with exit code 1
real 0m2.067s
rc=1

For ndots2, dnsConfig only added options ndots:2 to the cluster file. For dns-none, the file contains only what you wrote: no kube-dns, no ndots. This Pod therefore no longer resolves any Service name, and the query times out after 2 seconds. Add the ClusterIP of kube-dns to nameservers if this Pod must keep access to the names of the cluster.

How do you monitor and size CoreDNS?

CoreDNS exposes Prometheus metrics on port 9153 of the kube-dns Service. The port-forward makes them readable without a collector; the sleep 3 gives the tunnel time to open before curl.

Metrics and monitoring

Terminal window
kubectl -n kube-system port-forward svc/kube-dns 9153:9153 &
sleep 3
curl -s http://localhost:9153/metrics | grep -m5 coredns_dns_requests_total
kill %1
Forwarding from 127.0.0.1:9153 -> 9153
Forwarding from [::1]:9153 -> 9153
Handling connection for 9153
# HELP coredns_dns_requests_total Counter of DNS requests made per zone, protocol and family.
# TYPE coredns_dns_requests_total counter
coredns_dns_requests_total{family="1",proto="udp",server="dns://:53",type="A",view="",zone="."} 10
coredns_dns_requests_total{family="1",proto="udp",server="dns://:53",type="AAAA",view="",zone="."} 10
coredns_dns_requests_total{family="1",proto="udp",server="dns://:53",type="other",view="",zone="."} 1

The last three lines count the queries by record type: A, AAAA and others. Three other metrics complete the monitoring, according to the documentation of the metrics plugin and that of the cache plugin: coredns_dns_responses_total (answers by code, including NXDOMAIN and SERVFAIL), coredns_dns_request_duration_seconds (latency) and coredns_cache_hits_total (cache efficiency).

Scaling CoreDNS

The coredns Deployment has two replicas, and the section "The Pod resolves no name" showed that at zero, DNS is cut. To change the number of replicas (replace 2 with the value you want, and go back to 2 at the end of the lesson), or to restart the Pods:

Terminal window
kubectl -n kube-system scale deployment coredns --replicas=2
kubectl -n kube-system rollout restart deployment coredns

To adapt the number of replicas to the size of the cluster, the Kubernetes documentation describes a horizontal DNS autoscaler.

NodeLocal DNSCache: a DNS cache per node

For large clusters or applications that are sensitive to DNS latency, Kubernetes offers NodeLocal DNSCache, a DNS cache placed on every node by a DaemonSet. The setup is described in the official documentation.

Essential CKA commands

The CKA exam is timed: these commands cover the diagnostic path without writing anything. Pod names change, the k8s-app=kube-dns label does not.

Terminal window
kubectl get deployment,pods,svc -n kube-system -l k8s-app=kube-dns
kubectl get configmap coredns -n kube-system -o yaml
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
kubectl exec <pod> -- cat /etc/resolv.conf
kubectl exec <pod> -- nslookup <service>.<namespace>.svc.cluster.local
kubectl rollout restart deployment coredns -n kube-system

How do you clean up the lesson's cluster?

Delete the working namespaces, then check that CoreDNS is back to 2/2. The coredns ConfigMap lives in kube-system and survives this deletion: the last command checks that the Corefile no longer carries the hosts block or the 192.0.2.1 address of forward, and prints nothing if that is the case.

Terminal window
kubectl delete namespace ws-dns ws-dns-b
kubectl -n kube-system get deployment coredns
kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' | grep -E 'hosts|192\.0\.2\.1'

Put it into practice

A cluster DNS failure shows up everywhere at once and does not tell you its cause. In the guided lab, no Pod resolves a Service name any more: a team reports that its application in the app namespace no longer answers, although its web Pod, its web-svc Service and its client Pod have not changed. You find out why the cluster DNS no longer answers, you bring it back, then you prove that a client Pod resolves web-svc.app.svc.cluster.local and reaches the Service over HTTP.

The lab hides two faults. The first shows in the state of the Pods. The second is trickier: the CoreDNS Pods are Running and Ready, they look healthy, and yet the cluster names do not resolve. When the state of the Pods says nothing, you have to read the configuration that is loaded.

The validation reads the state of the cluster, not your commands: CoreDNS must run again as kubeadm sets it up, with two ready replicas and a configuration put back in order, then a real resolution and a real HTTP request start from the client Pod. A workaround inside the client Pod does not count. This lab runs on a kubeadm virtual machine, not on kind: the nodes are machines, and the CoreDNS configuration is the one that kubeadm installs.

Key takeaways

  • CoreDNS reads the Services and the EndpointSlices of the API and answers the Pods; it stores nothing. The /etc/resolv.conf of a Pod points to the kube-dns Service, and the CoreDNS Pods are selected by k8s-app=kube-dns.
  • The full name of a Service is <service>.<namespace>.svc.cluster.local; when diagnosing, always write it in full.
  • With pods insecure, a Pod name resolves even if no Pod holds the address: it is not a presence test.
  • The Corefile lives in the coredns ConfigMap of kube-system; keep a backup before changing it.
  • A change is not immediate (33 to 89 seconds on kind, allow two minutes), and a syntax mistake keeps the old configuration on the Pods that are running.
  • To restore a saved ConfigMap, remove resourceVersion from the file before replace, then restart CoreDNS.
  • An immediate Connection refused is not a network block: the Service has no Pod behind it, or the Pod does not listen on the targeted port. A timeout points to a path that does not answer.
  • A Pod with hostNetwork needs ClusterFirstWithHostNet to resolve the names of the cluster; dnsConfig adjusts or replaces its /etc/resolv.conf.

Test your knowledge

This quiz reuses the CoreDNS questions of the CKA program, where DNS troubleshooting belongs to the Troubleshooting domain. A score below 80% points to the sections to go back over.

Knowledge check

Check what you have learned with this interactive quiz

7 questions
5 min
80% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer