Applications across a node start timing out on DNS lookups. Then on database connections. Then on API calls. You check CoreDNS and the dashboards are green: low latency, no SERVFAILs, healthy pods. But the timeouts keep coming, and they are not limited to DNS.

This is conntrack table exhaustion. Every UDP DNS query that traverses Kubernetes iptables DNAT creates a connection tracking entry on the node with a default timeout of 30 seconds. At high DNS QPS, the table fills. When it does, the kernel drops new packets silently: no ICMP error, no reset, no log line the application will ever see. The kernel log says nf_conntrack: table full, dropping packet and nothing else anywhere warns you.

What makes this worse than a CoreDNS problem: conntrack is a shared, node-level resource. When it fills, the kernel drops new flows for everything on that node, not just DNS. Databases, message queues, the kubelet’s apiserver connection. The blast radius is the entire node, and CoreDNS is just the workload that usually fills the table first.

What this means

Linux netfilter tracks every connection through the node in the conntrack table so iptables can do NAT, including the DNAT that kube-proxy programs to send Service ClusterIP traffic (like kube-dns) to the right pod. UDP is “connectionless” to applications, but to netfilter each UDP flow is a tracked entry that lives for the unreplied-UDP timeout, 30 seconds by default.

That creates a simple capacity equation: a node doing 5,000 DNS QPS over UDP holds roughly 5,000 x 30 = 150,000 conntrack entries for DNS alone, before you count application TCP connections, health checks, or other UDP traffic. If nf_conntrack_max is below that steady-state demand, the table fills and the kernel drops.

The drops are silent by design. For a new UDP packet that cannot get a conntrack entry, the kernel simply discards it. The sender gets no feedback and waits for its own timeout. CoreDNS never sees the packet, so coredns_dns_requests_total does not move, latency looks excellent (the queries that do arrive are answered fast), and error rates are zero. Your CoreDNS metrics are accurate. They are measuring a server that stopped receiving traffic.

flowchart LR
  App["App pod
DNS query over UDP"] --> IPT["iptables DNAT
kube-dns ClusterIP"]
  IPT --> CT{"conntrack
table"}
  CT -->|"entry created"| CD["CoreDNS pod
answers normally"]
  CT -->|"table full: DROP"| Lost["packet silently dropped
client times out"]
  CT -->|"table full: DROP"| Other["ALL new flows on node
DB, API, kubelet also dropped"]

Common causes

CauseWhat it looks likeFirst thing to check
High DNS QPS exceeding table capacityConntrack count grows with query rate, plateaus at max, drops beginCompare nf_conntrack_count trend against CoreDNS QPS trend
Table sized too small for the nodeCount sits near max during normal hours, drops appear at peaksnf_conntrack_count / nf_conntrack_max ratio on busy nodes
Stale entries from pod restarts and connection churnCount stays elevated even when QPS is normalEntry creation rate vs expiry rate; correlate with pod churn events
kube-proxy conntrack cleanup issuesEntries pointing at long-deleted pod IPs linger; count climbs over daysInspect table with conntrack -L for entries to dead pod IPs
Non-DNS consumers on the same nodeConntrack fills on nodes that do not run CoreDNS at allBreak down table contents by destination port and protocol

Quick checks

All read-only and safe on an affected node.

# Current usage vs limit
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

# The definitive smoking gun: kernel log drops
dmesg | grep "nf_conntrack: table full"

# Conntrack subsystem counters, including insert failures
cat /proc/net/stat/nf_conntrack

# Per-CPU conntrack stats via the conntrack tool, shows insert_failed
conntrack -S

# Confirm CoreDNS itself is fine (expect low latency, low errors,
# and possibly a suspiciously low or flat query rate)
curl -s http://<coredns-pod-ip>:9153/metrics | grep '^coredns_dns_requests_total'

Two interpretation rules:

  • nf_conntrack: table full in dmesg is confirmation, not a clue. If you see it, packets were dropped. Treat it as an active incident on that node even if it has stopped printing (the message is rate-limited).
  • A CoreDNS query rate that dropped or flattened while clients report timeouts is itself a signal. UDP packets dropped by the kernel never reach the process and are never counted.

How to diagnose it

  1. Confirm the symptom is node-scoped. Check whether the failing applications share a node. If DNS timeouts, database connection errors, and API client retries all correlate to one or a few nodes, think shared node resource before thinking DNS.

  2. Check the table. Pull nf_conntrack_count and nf_conntrack_max on the suspect node. At or near 100% of max, you have your answer. Above 80%, you are one burst away.

  3. Check the kernel log. dmesg | grep "nf_conntrack: table full". This line closes the diagnosis. Absence does not fully clear conntrack: ring buffers rotate, and the separate conntrack race condition (intermittent 5-second DNS timeouts) happens below the full-table threshold. But full-table drops are ruled out.

  4. Verify CoreDNS is innocent. Scrape the CoreDNS metrics endpoint directly. Expect: latency low, SERVFAIL near zero, request rate lower than expected demand. This “too good to be true” profile, combined with client-side timeouts, is the classic signature of kernel-level drops before the process.

  5. Quantify the DNS contribution. Steady-state DNS conntrack entries are approximately UDP DNS QPS x 30 seconds (the default unreplied UDP timeout). If that number alone is a large fraction of nf_conntrack_max, DNS is your filler.

  6. Look at the rest of the table. If conntrack tools are available, conntrack -L shows what the entries actually are. On busy nodes you will often find the table shared between DNS flows and high-churn application connections; either one can be the majority consumer.

  7. Check for staleness. If the count stays high while traffic is normal, look for entries pointing at pod IPs that no longer exist. kube-proxy conntrack cleanup is best-effort, and stale UDP entries sit in the table burning capacity until they age out.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
node_nf_conntrack_entries / limit ratio (node_exporter)Direct measure of table headroomAbove 80% of limit; alert there, page at 90%
Kernel log nf_conntrack: table fullDefinitive confirmation of active dropsAny occurrence
insert_failed counters in /proc/net/stat/nf_conntrack or conntrack -SCounts drops even when you miss the dmesg windowAny sustained increment
CoreDNS coredns_dns_requests_total rateUnexpected drop with healthy process metrics means kernel-level lossRate falls while client DNS demand is known to be stable
CoreDNS latency and SERVFAIL rateRules CoreDNS in or out; expect both to look healthy during conntrack exhaustionGreen metrics plus client timeouts is the signature
Application-side DNS timeout rateThe only place the symptom is actually visibleIntermittent timeouts clustered at resolver retry intervals

Fixes

Emergency: raise the ceiling

# Raise the table limit immediately (safe, runtime change)
sysctl -w net.netfilter.nf_conntrack_max=524288

This stops the bleeding in seconds. It is a runtime change and does not survive reboot, so persist it through your node’s sysctl configuration afterward. Size it for steady state: expected UDP DNS QPS x 30 seconds, plus headroom for all other node traffic, then double it.

Note that kube-proxy manages conntrack sysctls on startup (--conntrack-max-per-core, default 32768, and --conntrack-min, default 131072) and can overwrite values you set elsewhere. If your sysctl keeps reverting, check the kube-proxy ConfigMap’s conntrack section and align it with your intended limit.

Reduce entry lifetime for UDP

# Expire unreplied UDP entries faster (default is 30s)
sysctl -w net.netfilter.nf_conntrack_udp_timeout=10

DNS flows are request-response and complete in milliseconds; 30 seconds of table residency per query is pure waste. Cutting it to 10 seconds reduces the DNS steady-state footprint by about 3x. Tradeoff: long-lived unreplied UDP flows (rare, but some tunneling and streaming protocols look like this) will lose NAT state and can break. Validate against what else runs on the node.

Long-term: take DNS off the conntrack path

The structural fix is NodeLocal DNSCache. It runs a caching agent as a DaemonSet on every node; pods query it over a link-local address, which bypasses the iptables DNAT and conntrack path, and the agent talks to upstream CoreDNS over TCP. DNS stops generating per-query UDP conntrack entries at all.

This also fixes the related conntrack race that causes intermittent 5-second DNS timeouts for parallel A/AAAA queries, which the table-raise does not. The tradeoff is operational: every node now runs a DNS layer you must monitor, and CoreDNS metrics reflect only cache misses from the local agents, not total cluster DNS demand.

Separate concerns

If you cannot deploy NodeLocal DNSCache, reduce co-location risk: keep CoreDNS pods and connection-churn-heavy workloads from concentrating on the same nodes, and make sure every node’s nf_conntrack_max is sized for the worst node, not the average one.

Prevention

  • Alert at 80% of nf_conntrack_max using the node_exporter conntrack entries metric against the limit. Conntrack exhaustion is a cliff with no graceful degradation; you need the warning before the edge.
  • Treat nf_conntrack: table full as a pageable event wherever kernel logs are collected. It means drops happened, full stop.
  • Capacity-plan conntrack like any other resource. Entries from DNS alone are roughly UDP QPS x 30s. Recompute when you add workloads or when ndots amplification changes your effective QPS.
  • Deploy NodeLocal DNSCache on clusters with meaningful DNS volume. It removes the entire failure class rather than raising the ceiling on it.
  • Include node conntrack state in CoreDNS incident runbooks. The first thing most teams check is CoreDNS, and CoreDNS will look healthy. The runbook should say: if clients time out but CoreDNS is green, check conntrack before touching CoreDNS.
  • Correlate after kube-proxy or CoreDNS upgrades. Cleanup regressions and stale entries tend to show up in the days after upgrades, as a slow climb in table usage rather than a spike.

How Netdata helps

  • Node-level conntrack visibility next to application metrics. Netdata collects per-node system metrics at one-second granularity, so you can watch conntrack utilization climb in real time instead of discovering it from a dmesg artifact after the fact.
  • The “green dashboard” correlation. The diagnostic moment in this incident is seeing CoreDNS request rate, latency, and error rate flat and healthy in the same view as node conntrack saturation and client-side timeouts. That juxtaposition redirects you from the DNS server to the kernel.
  • Node-scoped blast radius mapping. When conntrack fills, every workload on the node degrades together. Per-node views of network, socket, and application errors let you confirm the failure is node-scoped in one look, which is the fastest route to the right layer.
  • Threshold alerting before the cliff. Alerting on conntrack utilization crossing 80% gives you runway to raise the limit or shed load before any packet is dropped, which matters because the failure itself is instantaneous and silent.
  • Post-incident forensics. Because drops leave almost no trace in application logs, second-by-second history of conntrack count, CoreDNS request rate, and per-node network errors is often the only way to reconstruct when drops started and what filled the table.