CoreDNS UDP buffer errors: kernel receive-buffer overflow dropping queries

Applications are reporting intermittent DNS timeouts. You open the CoreDNS dashboard and everything looks fine: latency is low, SERVFAIL rate is zero, the health endpoint returns 200, and QPS is suspiciously flat. Not spiking, not zero, just lower than the client demand you know exists.

This is the UDP buffer cliff. Incoming UDP traffic is exceeding the kernel socket receive buffer on the node, and the kernel is dropping DNS queries before CoreDNS ever reads them. Because CoreDNS only counts the packets it actually receives, every CoreDNS-level metric looks clean. The failure lives one layer below, in the kernel.

This is an infrastructure problem, not a CoreDNS application problem. No Corefile change fixes it. The fix is at the node level: net.core.rmem_max and net.core.rmem_default, plus understanding why the buffer filled in the first place.

What this means

Every UDP DNS query lands in a kernel receive buffer attached to the CoreDNS listening socket. CoreDNS drains that buffer by reading packets from the socket. If packets arrive faster than CoreDNS reads them, the buffer fills. Once full, the kernel silently discards new datagrams. There is no ICMP error, no RST, no notification to the sender. The query vanishes, and the client waits out its full DNS timeout before retrying.

The default UDP receive buffer on many Linux distributions is 212992 bytes (about 208 KB). A single DNS query is roughly 50 to 500 bytes on the wire, so the default buffer holds on the order of hundreds to a few thousand queued queries. That sounds like a lot until you do the math: at 50,000 QPS, a 10 ms stall in packet processing (a GC pause, a CFS throttling window, a CPU-contended node) is enough to overflow it.

The distinguishing symptom shape is “too good to be true”:

• DNS query rate drops or stays flat, because dropped packets are never counted by CoreDNS.
• SERVFAIL rate is zero or low, because CoreDNS never saw the dropped requests.
• Request latency is low, because the queries that do arrive are served normally.
• Client-side DNS timeouts rise, with no matching CoreDNS error signal.

The last piece is what makes this a page-worthy incident: user-impacting DNS failures with no CoreDNS-level signal at all.

flowchart LR
  C[Client query burst] --> K[Kernel UDP receive buffer]
  K -->|buffer has room| CD[CoreDNS reads and answers]
  K -->|buffer full| D[Kernel drops packet silently]
  D --> T[Client waits for DNS timeout]
  CD --> M[CoreDNS metrics look clean]

Two related failure modes look similar from the client side and must be ruled out:

• Conntrack table exhaustion (Kubernetes): the kernel drops packets because nf_conntrack is full, with nf_conntrack: table full, dropping packet in the kernel log. Same silent-drop shape, different layer.
• The bufsize plugin and truncation issues: bufsize controls the EDNS0 buffer size CoreDNS advertises (default 1232 bytes). That is a DNS protocol buffer, completely independent of the kernel socket receive buffer. Do not confuse “dns: overflow unpacking uint16” style protocol errors with kernel RcvbufErrors; they have different causes and different fixes.

Common causes

Quick checks

All read-only and safe to run during an incident. Run these on the node hosting the CoreDNS pod (or on the CoreDNS host directly for non-Kubernetes deployments).

# Cumulative UDP receive buffer errors (the primary confirmation)
netstat -su | grep -i "buffer errors"

# Same data from procfs: RcvbufErrors and SndbufErrors columns
cat /proc/net/snmp | grep Udp

# Live counter check across UDP error types
nstat | grep -E "UdpInErrors|UdpRcvbufErrors|UdpSndbufErrors"

# Per-socket drops: find the CoreDNS socket (port 53 = hex 0035)
cat /proc/net/udp

# Extended per-socket view including drops
ss -u -a -e | grep ':domain'

# Current buffer limits
sysctl net.core.rmem_max net.core.rmem_default

Two sampling notes:

• These counters are cumulative. A single reading is only meaningful if it is nonzero and you know the host’s history. Take two readings 30 to 60 seconds apart; an incrementing RcvbufErrors during the incident window is the smoking gun.
• Be careful with tcpdump as a verification tool: AF_PACKET capture sockets have their own ring buffer and can drop packets under load, which can create misleading evidence either way.

How to diagnose it

1. Establish the symptom shape. Confirm CoreDNS reports low latency and zero SERVFAIL while clients report DNS timeouts. If CoreDNS is also showing SERVFAIL or high latency, you are likely dealing with a different failure (upstream problems, cache collapse), not the buffer cliff.

2. Check node-level UDP buffer errors. Run netstat -su | grep -i "buffer errors" and cat /proc/net/snmp | grep Udp on the node. RcvbufErrors incrementing during the incident window confirms packets are dropped at the socket buffer before reaching CoreDNS. This is the definitive check for this pattern.

3. Confirm it is per-socket. Check /proc/net/udp (drops column) or ss -u -a -e for the socket bound to port 53. Per-socket drops on the CoreDNS socket isolate the problem to the CoreDNS receive path rather than a host-wide UDP issue.

4. Rule out conntrack exhaustion. Check cat /proc/sys/net/netfilter/nf_conntrack_count against nf_conntrack_max, and dmesg | grep "nf_conntrack: table full". Conntrack exhaustion produces the same client-visible timeouts and the same clean CoreDNS metrics, but the fix is different.

5. Find why the buffer filled. A full buffer means arrival rate exceeded drain rate, so one side changed:

   • Arrival side: query flood, retry storm, DDoS, search-domain amplification multiplying real lookups.
   • Drain side: CoreDNS CPU saturation, CFS throttling, GC pauses, or node-level CPU contention slowing packet processing.

6. Compare throughput against expected demand. CoreDNS QPS that is flat or below baseline while clients are timing out means queries are being lost upstream of the metrics. This inversion (healthy server metrics, unhealthy client experience) is the pattern’s signature.

Metrics and signals to monitor

The composite rule that pages: client-visible DNS timeouts AND RcvbufErrors incrementing AND CoreDNS latency/SERVFAIL clean. Any one of these alone is ambiguous; together they are conclusive.

Fixes

Raise the kernel receive buffer

Apply on the node, not inside the container. In Kubernetes, net.core.rmem_max is not a namespaced sysctl, so pods cannot set it themselves; it must be set on the host (directly, via a privileged DaemonSet, or via your node provisioning tooling).

# Raise UDP receive buffer limits (25 MB in this example)
sysctl -w net.core.rmem_max=26214400
sysctl -w net.core.rmem_default=26214400

Persist it (/etc/sysctl.d/ or node configuration management) so it survives reboots. Common production values range from a few MB up to tens of MB; pick a value that covers your worst-case burst and verify drops stop.

Two cautions:

• net.core.rmem_default sets the default receive buffer for every new socket on the host, not just CoreDNS. Setting it to tens of MB raises memory consumption per socket host-wide. If you want to limit the blast radius, raise rmem_max generously but keep rmem_default more conservative.
• A larger buffer buys burst headroom, not capacity. If CoreDNS fundamentally cannot keep up with the sustained arrival rate, a bigger buffer only delays the drops and increases queuing latency for the packets that do get through.

Fix the drain rate

If drops persist after raising the buffer, CoreDNS is not reading fast enough:

• CPU starvation: raise the pod CPU limit or remove CPU limits if CFS throttling is the cause. Throttling stalls packet processing in 100 ms scheduler windows, which is more than enough to overflow a buffer at high QPS.
• GC pauses: if drop bursts align with GC activity, address heap pressure (cache sizing, GOMEMLIMIT) to shorten and reduce pauses.
• Node contention: co-scheduling CoreDNS with CPU-hungry workloads causes scheduling delays invisible to CoreDNS’s own latency metric. Consider dedicated nodes or priority scheduling for CoreDNS.

Reduce the arrival rate

• Identify and fix client retry loops and applications that poll DNS without caching.
• Address search-domain amplification (ndots:5 expansion multiplying each logical lookup into several wire queries) for workloads that mostly resolve external names.
• For DDoS-driven floods, rate-limit or filter upstream of the node; kernel buffers are not a DDoS mitigation.

Prevention

• Set net.core.rmem_max proactively on every node that runs CoreDNS, before you need it. The degradation curve is a cliff, not a slope: buffer size must exceed worst-case burst rate times worst-case packet processing latency.
• Alert on RcvbufErrors increments at the node level. This counter should be zero in normal operation; any sustained increment is actionable.
• Alert on the inversion, not just on errors: CoreDNS QPS significantly below expected client demand during known-active hours, or client-side DNS timeout rates rising while CoreDNS metrics are clean.
• Monitor conntrack utilization on CoreDNS nodes alongside buffer errors, since both produce identical client symptoms.
• Correlate drops with GC and CPU signals in your dashboards so the drain-side cause is visible without manual digging during an incident.

How Netdata helps

Netdata’s value for this failure is that it sees both sides of the kernel/application boundary at once:

• Node-level UDP error counters (RcvbufErrors, SndbufErrors, InErrors from /proc/net/snmp) are collected per second, so drop bursts are visible as they happen rather than as a cumulative counter you have to poll manually.
• CoreDNS metrics (request rate, latency, SERVFAIL, GC duration) on the same dashboard as the node counters, making the “clean app metrics plus rising kernel drops” inversion immediately visible.
• CPU and CFS throttling correlation: per-second container CPU and throttling data next to the drop timeline shows whether CoreDNS was starved of CPU at the moment packets were dropped.
• Conntrack utilization on the same node view, so the two lookalike silent-drop failures can be distinguished without separate manual checks.
• Per-second granularity matters specifically here: buffer overflows are burst events lasting seconds, and minute-resolution monitoring can average them away entirely.

Related guides

• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS cache collapse: the cold-cache thundering herd after a rollout
• CoreDNS cache hit ratio dropping: latency and upstream load climbing together
• CoreDNS CPU throttling: CFS limits making a green dashboard lie about latency
• CoreDNS GC pauses adding tail latency: go_gc_duration_seconds and heap pressure
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed