CoreDNS 5-second DNS timeout: the Kubernetes glibc A+AAAA conntrack race

Your application latency histogram has a cluster of samples at almost exactly 5 seconds. Not 4.8, not 5.3: a tight spike right at the glibc default resolver timeout. Tracing the slow calls shows they stall on name resolution. CoreDNS dashboards are green: low latency, no SERVFAIL, healthy cache hit ratio, both replicas fine.

That combination, application-side DNS stalls at exactly the resolver timeout with clean CoreDNS metrics, is the signature of the glibc A+AAAA conntrack race. The loss is in the kernel network path between the application pod and CoreDNS, not inside CoreDNS. No amount of Corefile tuning will touch it.

What this means

The glibc resolver, which most containers use, sends A and AAAA queries in parallel from the same UDP socket when a process resolves a name. In Kubernetes, pod traffic to the kube-dns ClusterIP crosses a NAT boundary (iptables or IPVS DNAT through kube-proxy), so every packet passes through the kernel conntrack subsystem.

When two UDP packets leave the same socket at effectively the same time and no confirmed conntrack entry exists yet, a kernel race can drop one of the two packets during conntrack confirmation. The query never reaches CoreDNS, or its response never makes it back. There is no ICMP error, no reset, no SERVFAIL. The application hears nothing for that query and waits out its resolver timeout, 5 seconds by default, before retrying on a fresh socket. The retry usually succeeds, which is why the failure is intermittent.

The race is probabilistic. Its probability rises with conntrack pressure on the node: more concurrent flows, more table churn, more simultaneous first-packet events. That is why it shows up on busy nodes first and disappears when you try to reproduce it in a quiet test environment.

flowchart LR
  app[Application pod] -->|A + AAAA queries, same UDP socket| sock[glibc resolver]
  sock -->|two packets at once| ct[conntrack / DNAT path]
  ct -->|race: one packet dropped| drop((lost query))
  ct -->|other packet delivered| cdns[CoreDNS]
  cdns -->|fast response, metrics clean| sock
  drop -.->|no response| timeout[app waits 5s, retries]

Two consequences matter for diagnosis:

1. CoreDNS never sees the dropped query. coredns_dns_requests_total does not count it, latency metrics do not include it, and there is no error anywhere in CoreDNS telemetry.
2. The 5-second value is not a network property. It is the client’s resolver timeout. If an app’s timeout is configured differently, the spike moves to that value. The spike location tells you which component is timing out.

Common causes

The distinguishing feature of the race, versus plain conntrack exhaustion, is intermittency. Table exhaustion drops packets deterministically once full and affects every protocol on the node. The race drops a small percentage of first-packet pairs continuously, even at moderate table utilization, and only hits UDP flows that race at creation time, which in practice means DNS.

Quick checks

All read-only. Run from a node shell (or a debug container on the node) unless noted.

# 1. Confirm the 5s spike is in application metrics, not server metrics.
#    Pull CoreDNS latency: if p99 is healthy (sub-50ms) while apps report 5s stalls,
#    the loss is in the path, not the process.
curl -s http://localhost:9153/metrics | grep 'coredns_dns_request_duration_seconds'

# 2. Check conntrack table pressure on nodes hosting affected pods
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

# 3. Rule out full-table exhaustion (different failure, different fix)
dmesg -T | grep -i "nf_conntrack: table full"

# 4. Check conntrack drop counters
cat /proc/net/stat/nf_conntrack

# 5. Rule out UDP buffer drops at the CoreDNS socket
cat /proc/net/snmp | grep Udp

# 6. Confirm which resolver the affected workloads use.
#    From inside an affected pod:
cat /etc/resolv.conf
#    Look for timeout/attempts/ndots and any existing options lines.
#    Also confirm the image libc: glibc-based images exhibit this race;
#    Alpine/musl images have different resolver behavior.

# 7. Probe with timing from an affected pod (read-only lookup loop)
for i in $(seq 1 50); do dig example.com +tries=1 +stats | grep "Query time"; done
#    Watch for occasional timeouts (default 5s) amid mostly fast responses.

Note on check 7: dig issues one query type per invocation and uses its own resolver path, so it does not reproduce the parallel A+AAAA pattern that triggers the race. A clean run does not rule the race out. Application histograms are the primary evidence.

How to diagnose it

1. Establish the client-side signature. Pull the application’s DNS or connection-establishment latency histogram. You are looking for a bimodal distribution: a fast cluster in single-digit milliseconds and a second cluster at exactly the resolver timeout (5s for glibc defaults). A spike at a round timeout value is almost never a server-side processing problem; it is a client waiting for a packet that never arrives.

2. Verify CoreDNS is innocent. Check coredns_dns_requests_total, coredns_dns_request_duration_seconds, and coredns_dns_responses_total{rcode="SERVFAIL"}. In this failure mode all three look normal. Kernel-dropped packets are never counted by CoreDNS, so clean metrics are expected, not exculpatory by themselves. The combination of clean server metrics and client-side timeout spikes is the diagnostic.

3. Check conntrack headroom on the affected nodes. High nf_conntrack_count relative to nf_conntrack_max raises race probability and tells you how close you are to the worse failure, table exhaustion. Above 80%, treat conntrack pressure as a contributing cause regardless of the race.

4. Rule out the lookalikes. No “table full” in dmesg rules out exhaustion. No incrementing RcvbufErrors in /proc/net/snmp rules out socket buffer drops at CoreDNS. If SERVFAILs exist, you have a different problem; the race produces silence, not error responses.

5. Confirm the query pattern. The race requires concurrent A and AAAA from one socket. glibc-based multi-threaded apps doing fresh lookups (no local caching, short-lived processes, or heavy ndots:5 search-domain expansion multiplying query volume) are the prime candidates. If affected pods run Go binaries with the pure-Go resolver, or musl-based images, suspect a different cause.

6. Apply a mitigation on one workload and measure. Roll out single-request-reopen (see Fixes) to one affected Deployment. If the 5s spike disappears from that workload’s histogram while nothing else changed, you have confirmed the mechanism.

Metrics and signals to monitor

Fixes

Deploy NodeLocal DNSCache (the real fix)

NodeLocal DNSCache runs a caching DNS agent as a DaemonSet on every node and intercepts pod DNS traffic before it crosses the iptables DNAT boundary. That removes the conntrack/NAT hop from the pod-to-DNS path entirely, which eliminates the race rather than reducing its probability. It also upgrades cache-miss traffic to TCP toward the cluster CoreDNS, sidestepping UDP conntrack behavior further, and offloads most query volume from cluster CoreDNS.

Tradeoffs: it is another component to run and monitor, one agent per node. After deployment, cluster CoreDNS QPS drops sharply because it only sees cache misses from the node-local layer. Do not mistake that drop for a problem, and do not stop monitoring cluster CoreDNS: upstream failures now surface only through the miss path.

Work around it in the pod’s resolver config

For workloads you cannot move to NodeLocal DNSCache immediately, change how glibc issues the parallel queries via dnsConfig in the pod spec:

spec:
  dnsConfig:
    options:
      - name: single-request-reopen

single-request-reopen makes glibc close the socket and open a fresh one before sending the second query, so the two packets no longer race on one socket’s conntrack state. The related option single-request serializes A and AAAA over the same socket instead. Both eliminate the race; both add latency to every dual-family lookup, because the AAAA answer now arrives later. That cost is far smaller than an occasional 5-second stall, but it is not zero, so treat this as mitigation, not architecture.

Caveats: these are glibc options. Alpine/musl images do not honor them the same way, and Go binaries built with the pure-Go resolver bypass glibc entirely, so neither option helps there. Per-pod dnsConfig also means rolling this out workload by workload, or via an admission webhook, which is operationally tedious at fleet scale.

Force TCP for DNS

The resolver option use-vc forces glibc to use TCP for DNS queries. TCP conntrack entries are established by the handshake rather than raced into existence by concurrent datagrams, so the race class disappears entirely. The cost is a full TCP handshake and connection overhead per lookup: measurably slower than UDP for every query, not just the unlucky ones. Prefer single-request-reopen unless you have a specific reason to want TCP, for example consistently large responses that truncate over UDP anyway.

Reduce conntrack pressure on the node

This does not fix the race but lowers its frequency and keeps you away from table exhaustion:

• Raise net.netfilter.nf_conntrack_max on busy nodes. A bare sysctl is reactive; persist it through node provisioning so it survives replacement.
• Shorten UDP conntrack entry lifetime: sysctl -w net.netfilter.nf_conntrack_udp_timeout=10 reduces the time dead DNS flows occupy the table (default 30 seconds).
• Reduce query amplification: ndots:5 turns one external lookup into several queries, each adding conntrack churn. Setting ndots to 1 for external-facing workloads, or using fully qualified names with a trailing dot, cuts DNS query volume 4-6x.

Upgrade the kernel

Upstream kernel fixes for the conntrack races behind this symptom landed in Linux 4.19 (commit ed07d9a021df, “netfilter: nf_conntrack: resolve clash for matching conntracks”) and Linux 5.0 (commit 4e35c1cb9460, “netfilter: nf_nat: skip nat clash resolution for same-origin entries”), with the second fix backported to stable releases 4.9.163, 4.14.106, 4.19.29, and 4.20.16. Nodes running older kernels, or vendor kernels without the backports, remain exposed. Check your node kernel version against your distribution’s advisory before assuming the fixes are present, and treat resolver workarounds as defense-in-depth even on modern kernels.

Prevention

• Run NodeLocal DNSCache on any cluster where DNS latency matters. It removes this entire failure class plus conntrack exhaustion risk for DNS, and absorbs query bursts at the node.
• Monitor conntrack utilization on every node, not just CoreDNS pods. The conntrack table is shared node infrastructure that no CoreDNS metric exposes. Alert on usage above 80% of max and on any table-full kernel message.
• Collect application-side DNS latency. CoreDNS-reported latency excludes kernel buffer waits, network transit, and dropped packets. If you only watch the server side, this failure is invisible by construction.
• Tame ndots:5 amplification for workloads that mostly resolve external names. Fewer queries means less conntrack churn, lower race probability, and lower CoreDNS load at once.
• Keep node kernels current within your distribution’s support window so the upstream conntrack fixes are actually running.

How Netdata helps

Netdata’s value here is correlation across the layers this failure crosses:

• Per-second node-level conntrack metrics show table utilization and churn on every node, surfacing the pressure that drives race frequency before exhaustion hits.
• CoreDNS Prometheus metrics (coredns_dns_requests_total, coredns_dns_request_duration_seconds, SERVFAIL rates) are collected alongside node metrics in one place, making the “clean server, broken client” signature visible in a single view instead of two dashboards.
• UDP error counters from /proc/net/snmp (RcvbufErrors and friends) are charted per node, so you can rule socket-buffer drops in or out without SSH-ing during an incident.
• Comparing application-reported latency against CoreDNS-reported latency at the same timestamps quantifies the gap that kernel-path drops create, which is the fastest route from “DNS is slow” to “DNS is slow between the pod and CoreDNS.”

Related guides

• CoreDNS cache hit ratio dropping: latency and upstream load climbing together
• CoreDNS CPU throttling: CFS limits making a green dashboard lie about latency
• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS cache collapse: the cold-cache thundering herd after a rollout
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed