Symptom

In a Kubernetes cluster, CoreDNS is used to provide internal DNS services. However, during operation, occasional failures in resolving public domain names were observed. Application logs recorded UnknownHost errors.

Root Cause Analysis

Network packet capture was performed on the problematic cluster. When DNS resolution failures were logged on the service side, the DNS resolution records revealed the following:

• DNS ultimately completed resolution and returned the correct result to the service.
• The service threw an UnknownHost error before receiving the final correct resolution result.
• When the service requested DNS resolution for a domain name, it did not directly query the full domain name. Instead, it appended Kubernetes internal domain suffixes (e.g., .svc.cluster.local) for resolution.

Explanation

The root cause lies in the DNS resolution priority within the Kubernetes cluster.

Every Pod in a Kubernetes cluster has an /etc/resolv.conf configuration file. A typical example looks like this:

search c-moon.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.10.10.5
options ndots:2

Two key parameters in this file are search and ndots.

• search: A list of domain names (hostanmes) to search. Currently limited to 6 domains and 256 characters total.
• ndots: Represents the number of dots (.) in a domain name.

Key behavior:

• If a domain name is not a fully qualified domain name (FQDN) and contains fewer dots than the ndots value, the reslover will first attempt to resolve it by appending each domain from the search list in order.
• If a domain name is not an FQDN but contains dots equal to or greater than the ndots value, the resolver will resolve the domain name as-is (without appending search domains).

PS - Fully Qualified Domain Name (FQDN): A domain name ending with a dot (.). For example, baidu.com is NOT an FQDN, but baidu.com. IS an FQDN.

Example

If you need to resolve baidu.com within the cluster, given the /etc/resolv.conf configuration above, the actual resolusion sequence is:

1. baidu.com.c-moon.svc.cluster.local
2. baidu.com.svc.cluster.local
3. baidu.com.cluster.local
4. baidu.com

Solution

Once the cause was understood, two approaches were considered:

1. Increase the DNS resolution timeout on the service side.
2. Modify the ndots parameter to reduce unnecessary DNS queries.

Modifying the ndots Parameter

The default ndots value is 1, but CoreDNS changes it to 5. Instead of modifying CoreDNS globally, we can configure the ndots value specifically for a Deployment (DP).

spec:
  dnsConfig:
    options:
      - name: ndots
        value: '2'
  containers:
    - image: nginx
      imagePullPolicy: Always
      name: nginx

References

• DNS for Services and Pods
• Discussion on resolv.conf search and ndots configuration