BIND UDP receive buffer errors: the invisible query loss named never logs

Clients report intermittent DNS timeouts. Retries are up. Your monitoring says named is healthy: the process is running, port 53 is open, functional queries succeed, and BIND’s statistics channel shows reasonable response rates with no elevated SERVFAIL. You suspect upstream nameserver problems, network path issues, or client-side misconfiguration. None of those investigations turn up anything.

The problem may be happening between the kernel and BIND, in a layer where named has zero visibility. When the kernel’s UDP receive buffer overflows, packets are silently dropped before BIND ever reads them from the socket. There is no log entry, no statistics counter increment, no error of any kind inside named. The only evidence lives in kernel-level counters that most BIND monitoring setups never collect.

The symptom pattern is distinctive but easy to misdiagnose: random client timeouts, elevated retry rates, and a BIND instance that looks perfectly healthy from its own instrumentation.

What this means

BIND’s query processing pipeline begins when a worker thread reads a UDP datagram from the socket. Everything before that read is kernel territory. The kernel receives packets from the NIC, places them in the socket’s receive buffer (a fixed-size kernel memory queue), and waits for the application to drain it. If the buffer fills before BIND reads fast enough, the kernel drops new arrivals and increments the UdpRcvbufErrors counter in /proc/net/snmp.

BIND’s statistics only count queries it successfully read from the socket. Dropped-at-kernel-level queries produce no increment in any BIND counter: no QryUDP, no QrySERVFAIL, no Requestv4. They simply never existed from named’s perspective. This is why the server looks healthy while clients are experiencing packet loss.

The failure is asymmetric in a way that makes diagnosis harder. Bandwidth utilization may be low because this is a packets-per-second limit, not a bytes-per-second limit. DNS packets are small. CPU may look moderate in aggregate. The bottleneck is often on a single core processing network softirqs, hidden behind healthy-looking aggregate numbers. In anycast deployments, the problem is frequently local to one node, masked by stable global aggregates from other nodes that are absorbing traffic fine.

flowchart LR
  A[Client query] --> B[Kernel NIC ring]
  B --> C[Socket recv buffer]
  C -->|BIND reads in time| D[named processes query]
  C -->|Buffer full| E[Kernel drops packet]
  E --> F[UdpRcvbufErrors++]
  E -.->|No log, no counter| G[BIND never sees it]
  G --> H[Client timeout + retry]
  H -.->|Misdiagnosed as| I[Upstream problem?]

Common causes

Quick checks

All commands below are safe and read-only. Run them on the node exhibiting symptoms.

# Check kernel UDP receive buffer error counter (cumulative since boot)
cat /proc/net/snmp | grep -w Udp

# Or use netstat for a human-readable summary (if available; deprecated on some distros)
netstat -su 2>/dev/null | grep -i "receive buffer errors"

# Per-socket drops. On modern kernels, the last column is drops (requires gawk for strtonum).
# Check /proc/net/udp6 as well if the server handles IPv6 traffic.
awk 'NR>1 {split($2,a,":"); if ($NF>0) print "port=" strtonum("0x" a[2]), "drops=" $NF}' /proc/net/udp

# Current socket buffer size limits
sysctl net.core.rmem_max net.core.rmem_default

# Check socket receive queue depth for named's UDP listeners
ss -unlp | grep named

# Per-core CPU and softirq distribution (5 samples at 1-second intervals)
mpstat -P ALL 1 5

# BIND's own view: verify the query rate BIND thinks it is receiving.
# Default statistics channel port is 8053; varies by config.
curl -s http://localhost:8053/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); ns=d.get('nsstats',{}); \
  print('Requestv4:', ns.get('Requestv4',0), 'Requestv6:', ns.get('Requestv6',0), \
  'QryUDP:', ns.get('QryUDP',0))"

The key diagnostic signal is a non-zero UdpRcvbufErrors rate during production traffic. This counter is cumulative since boot, so you need to sample it twice and compute the delta. Any sustained non-zero rate is abnormal for a DNS server.

How to diagnose it

1. Confirm the drops exist. Sample UdpRcvbufErrors twice, 10 seconds apart. If the value increased, packets are being dropped at the kernel level.

   # Sample 1
   cat /proc/net/snmp | grep -w Udp | tail -1
   sleep 10
   # Sample 2 - compare the UdpRcvbufErrors column
   cat /proc/net/snmp | grep -w Udp | tail -1
   

2. Check whether UdpInErrors is also rising. UdpRcvbufErrors is generally a subset of UdpInErrors. If UdpInErrors is increasing but UdpRcvbufErrors is zero, the problem is packet corruption (checksum errors tracked as UdpInCsumErrors in the same /proc/net/snmp line), not buffer overflow. The fix path is entirely different.

3. Verify BIND’s own query counters are not showing the lost traffic. Compare the rate of Requestv4 plus Requestv6 from BIND’s statistics channel against the actual inbound packet rate on the interface. A gap means packets are being lost before BIND sees them. This gap is the invisible query loss.

4. Identify the bottleneck. Check per-core CPU with mpstat -P ALL 1 5. Look for one or two cores at high softirq (the %soft column) while others are idle. This indicates a single-core packet processing bottleneck.

5. Check current socket buffer sizes. Run sysctl net.core.rmem_max and sysctl net.core.rmem_default. The Linux defaults are commonly too small for high-QPS DNS servers. rmem_max is the ceiling the kernel enforces regardless of what the application requests. rmem_default is the initial buffer allocated to each new socket.

6. Verify anycast isolation. If you run anycast, check each node individually. The problem is often local to one node, masked by stable global aggregates from other nodes that are handling the traffic without issue.

Metrics and signals to monitor

Fixes

Increase socket buffer sizes

The most direct fix is to increase the kernel’s maximum receive buffer size. This gives BIND (and the kernel) more headroom to absorb bursts.

# Check current values
sysctl net.core.rmem_max net.core.rmem_default

# Raise the ceiling at runtime (example value for a high-traffic resolver).
# This only takes effect for sockets created or reconfigured after the change;
# restart named to ensure its listener sockets pick up the new limit.
sysctl -w net.core.rmem_max=33554432

# Optionally raise the default initial allocation for all new sockets.
# Keep this modest: it applies system-wide, not just to BIND.
sysctl -w net.core.rmem_default=4194304

These are runtime changes. To persist across reboots, add them to /etc/sysctl.d/ or your distribution’s sysctl configuration.

The kernel caps the socket buffer at net.core.rmem_max regardless of what the application requests. Even if BIND’s so-rcvbuf option requests a larger buffer, the kernel silently caps it. You must raise the sysctl alongside any BIND-side buffer configuration. If your BIND was compiled with --with-tuning=large, the internal buffer constants are already larger, but this only affects what BIND requests from the kernel. The sysctl ceiling still applies.

Tradeoff: larger buffers consume more kernel memory per socket. Setting rmem_default high applies to every socket on the system, not just BIND’s listeners. For a dedicated DNS server with a small number of listeners, the cost is negligible. On mixed-workload hosts, keep rmem_default modest and raise only rmem_max, then configure BIND’s so-rcvbuf to request the buffer it needs.

Resolve the single-core bottleneck

Increasing buffer size buys time but does not fix a packet-processing throughput problem. If one core is saturated handling network softirqs, larger buffers just delay the drops slightly. You need to distribute the load.

Check current IRQ distribution for your network interface:

# View interrupt counts per CPU for your NIC
cat /proc/interrupts | grep -i <interface-name>

# Check which CPUs each RX queue's IRQ is affined to
for irq in $(grep <interface-name> /proc/interrupts | cut -d: -f1); do
  echo "IRQ $irq: $(cat /proc/irq/$irq/smp_affinity_list 2>/dev/null)"
done

If all RX queues are affined to the same CPU, redistribute them across multiple cores. The exact mechanism depends on your NIC driver and whether you use irqbalance, manual /proc/irq/*/smp_affinity tuning, or NIC-specific tools like ethtool -L for multichannel configurations.

BIND 9.12+ supports SO_REUSEPORT, which creates multiple UDP sockets per listener and spreads receive processing across worker threads. This helps when the bottleneck is single-socket contention rather than pure IRQ saturation. The -U command-line option controlled the number of UDP dispatches in older versions, but it was removed in BIND 9.20.0. On 9.20+, the dispatch behavior is managed internally.

Rule out NIC-level drops

If UdpRcvbufErrors is zero but clients still report timeouts, the drops may be happening at the NIC level before packets reach the kernel UDP stack. Check interface counters:

ethtool -S <interface> | grep -i drop

NIC ring buffer overflows produce similar symptoms but require a different fix (increasing ring buffer size via ethtool -G). This is a distinct problem from socket buffer overflow, though both can coexist under heavy packet load.

Prevention

• Monitor UdpRcvbufErrors from day one. This is a kernel counter, not a BIND counter. It must be collected at the OS level, not from the statistics channel. Any sustained non-zero rate during production traffic is abnormal.
• Set net.core.rmem_max proactively. Do not wait for drops to appear. A high-traffic DNS server needs buffer headroom that Linux defaults do not provide.
• Track per-core CPU softirq. A single hot core is the leading indicator of a packet-processing bottleneck. Catching this before drops begin gives you time to fix IRQ affinity.
• Monitor per-node in anycast deployments. Aggregate anycast metrics hide single-node saturation. Each node needs its own kernel counter collection.
• Collect Recv-Q for named’s UDP sockets. Persistent non-zero Recv-Q means BIND is not draining the socket fast enough, and drops are imminent or already happening.

How Netdata helps

Netdata collects kernel-level UDP statistics, including UdpRcvbufErrors, at per-second resolution directly from /proc/net/snmp. This is a signal that BIND’s own statistics channel cannot provide.

• Per-second UdpRcvbufErrors collection means you see drops the moment they start, not minutes later when a slower polling interval catches up.
• Correlating UdpRcvbufErrors with per-core CPU softirq (also collected per-second) immediately distinguishes buffer-too-small from single-core bottleneck, guiding you toward the right fix.
• Correlating kernel drops with BIND’s Requestv4 and Requestv6 counters reveals the gap between packets arriving and packets BIND actually processed. A widening gap is the invisible query loss made visible.
• Anomaly detection on UdpRcvbufErrors flags the transition from zero to non-zero without requiring manual threshold tuning. The normal state for this counter is flat zero. Any deviation is actionable.
• Per-node dashboards in anycast deployments prevent one saturated node from hiding behind healthy aggregate metrics.

Related guides

• BIND DNSSEC validation failing: ‘broken trust chain’, ValFail, and SERVFAIL for signed domains
• BIND cache eviction storms: DeleteLRU, an undersized max-cache-size, and the pressure spiral
• BIND cache hit ratio dropping: the leading edge of recursive pain
• BIND clients-per-query and max-clients-per-query: duplicate recursion for popular names
• BIND cold cache after restart: the warming storm and elevated upstream load
• BIND DNSSEC failing from clock drift: NTP, RRSIG inception/expiry windows, and SERVFAIL
• BIND dynamic update failures: UpdateFail, denied updates, and TSIG drift
• BIND forwarding loops: recursion that never terminates and burns recursive slots
• How BIND actually works in production: a mental model for operators
• BIND journal (.jnl) corruption: dynamic-update and IXFR failures that block zone load
• BIND lame delegations: ’lame server resolving’ and nameservers that are not authoritative
• BIND managed-keys and trust anchors: KSK rollover, RFC 5011, and a stale root key