DNS resolution is one of those things that works as expected for the most port. This combined with the fact that the weeds of DNS resolution are abstracted out by the higher level languages leads to a situation where most developers don’t have a good idea how DNS works.
Having a good understanding of how DNS works and the tools to explore the mysteries of DNS resolution will help developers debug connectivity issues better before reaching out to SREs and DevOps teams.
To keep it more exciting why don’t we look at DNS in Kubernetes. Let’s put DNS in Kubernetes under a (developer) microscope and look at all its innards. What fun..
Kubernetes uses CoreDNS for Service Discovery using DNS
I have configured CoreDNS to log all the DNS queries its getting and the responses. This is not recommended for production deployments, but very handy to learn how things work.
From an alpine pod running in the Kubernetes cluster I did nslookup yahoo.com
yahoo.com obviously is a domain name that can’t be resolved by CoreDNS within this cluster. What happened here ? Let’s look at the CoreDNS logs.
What triggered the first four resolutions that failed before eventually yahoo.com succeeded ? To understand this we should look at the /etc/resolv.conf file in the pod
Unix/Linux systems have a stub resolver which is a thin DNS client, part of the process, to handle the queries. /etc/resolv.conf has the configurations used by the resolver.
When the pod is executed Kubernetes sets up the /etc/resolver.conf file with the desired config. Since we want CoreDNS to be resolver its configured as the only nameserver; 172.20.0.10 is the ClusterIP of the kube-dns (its still named kube-dns which I believe was done for seamless migration from using kube-dns to CoreDNS).
All this is good, but why we do see those extra resolution attempts. That has got to do with the ndots:5 configuration
What this means is if the user entered query has 5 or more dots it is sent as is to the upstream resolver. Query name user entered is considered the absolute name. Since yahoo.com has only one “.” the resolver should first try by appending every one of the subdomain names listed for the search option. Kubernetes sets up the resolv.conf with these subdomains in the search so that app developers can get unqualified names resolved within the cluster. For the basics on how DNS resolution works in Kubernetes refer to this official documentation. In this case the pod is running in a Kubernetes namespace non-istio.
Ok, we got some answers so far but we ended up more questions as well…
- How did the name yahoo.com actually get resolved ?
- If my app makes lot of DNS queries how can I avoid these extra queries ?
- How does CoreDNS keep track of the IPs of all services being deployed ?
If we step outside the realm of Kubernetes there are more interesting questions
- Why there is no corresponding AAAA record query (Its to get the IPv6 address) for the first four A record queries ?
- Do we really have to resolve the domain name to both A and AAAA; isn’t that unnecessary if I don’t care about either IPv4 or IPv6
We will cover these topics in the upcoming articles in this series.
