BIND UDP packet-rate saturation: softirq, single-core bottlenecks, and pps limits

DNS queries are timing out for some clients. BIND statistics look clean: query rate is steady, SERVFAIL is low, cache hit ratio is normal. The link is not saturated. BIND logs show nothing unusual. But clients keep reporting intermittent failures with no apparent correlation to domain, client subnet, or time of day.

If you have ruled out upstream issues, cache pressure, and DNSSEC failures, the problem may be in the kernel. When the host cannot process UDP packets fast enough, the kernel drops them before BIND reads them from the socket. BIND has no visibility into these drops: no log entry, no statistics counter, no error. The only evidence lives in kernel-level UDP counters that most DNS monitoring setups never collect.

This is a packets-per-second (pps) limit, not a bandwidth limit. DNS queries are small UDP datagrams, typically 60 to 512 bytes. A server processing 100,000 queries per second at 200 bytes each moves only about 160 Mbps of payload. Bandwidth reads as trivial while the packet processing path is saturated.

In anycast deployments, drops are often local to one node while the global aggregate looks stable. Clients routed to the affected node see timeouts and retries; clients hitting healthy nodes see no problem. This makes the issue easy to miss in aggregated monitoring.

How the kernel receive path works

The Linux kernel UDP receive path has two stages:

1. The NIC raises a hardware interrupt when packets arrive. The interrupt handler runs on a specific CPU core and schedules a softirq (software interrupt) to process the packet. The softirq copies the packet from the NIC ring buffer into the kernel socket receive buffer.
2. BIND reads packets from the socket buffer.

If the softirq cannot drain the NIC ring buffer fast enough, or if BIND cannot drain the socket buffer fast enough, packets accumulate and the kernel drops the overflow. Dropped packets increment UdpRcvbufErrors in /proc/net/snmp. BIND never sees them.

flowchart TD
    A["Inbound UDP packets"] --> B["NIC hardware interrupt on CPU N"]
    B --> C["Softirq on same core drains NIC ring"]
    C --> D["Kernel queues to UDP socket buffer"]
    D --> E{"Socket buffer full?"}
    E -->|"No"| F["BIND reads and processes query"]
    E -->|"Yes"| G["Kernel drops packet"]
    G --> H["UdpRcvbufErrors increments"]
    H --> I["No BIND log or counter"]
    I --> J["Client sees random timeout"]

Two mechanisms drive these drops:

Single-core softirq saturation. Without Receive Side Scaling (RSS), all NIC interrupts land on one core (typically CPU 0). That core’s softirq saturates while other cores sit idle. Aggregate CPU utilization looks moderate, but one core is pinned at 100% in the softirq component.

Socket buffer exhaustion. If BIND’s worker threads cannot read from the socket fast enough (busy with query processing, DNSSEC validation, or RPZ evaluation), the socket receive buffer fills and the kernel drops new arrivals. On most Linux distributions, net.core.rmem_default is 212992 bytes (208 KiB), which is small for a high-pps DNS server.

This distinction matters for diagnosis. If the bottleneck is in softirq processing, raising the socket buffer buys time but does not fix the root cause. If the bottleneck is in BIND’s consumption rate, raising the buffer is the direct fix. Both often need attention.

Common causes

Quick checks

All commands below are read-only and safe to run on a production server.

# Check kernel UDP receive buffer errors (cumulative since boot)
cat /proc/net/snmp | grep Udp

# More readable format for UDP error counters
nstat -s | grep -i udp

# Check per-socket drops on port 53 (drops is the last field; requires kernel 3.5+)
awk 'NR>1 && $2 ~ /:0035$/ {print $2, "drops="$NF}' /proc/net/udp
awk 'NR>1 && $2 ~ /:0035$/ {print $2, "drops="$NF}' /proc/net/udp6

# Check receive queue depth on named's UDP sockets
ss -unlp | grep ':53 '

# Per-core CPU utilization including softirq breakdown (5 samples, 1s interval)
mpstat -P ALL 1 5

# Check current socket buffer size limits
sysctl net.core.rmem_max net.core.rmem_default

# Check NIC receive queue count
ethtool -l eth0

# Check which CPUs handle NIC interrupts
cat /proc/interrupts | grep eth0

The most important single check is nstat -s | grep -i UdpRcvbufErrors (or the equivalent grep on /proc/net/snmp). If this counter is incrementing during production traffic, the kernel is dropping DNS packets.

How to diagnose it

1. Confirm packet drops exist. Sample UdpRcvbufErrors twice with a few seconds between samples. If the value increases, the kernel is actively dropping packets. Note that UdpInErrors is a superset that also includes checksum errors and NoPort drops. Focus on UdpRcvbufErrors specifically, as it indicates buffer exhaustion.

2. Check per-core softirq distribution. Run mpstat -P ALL 1 5 and look at the %soft column. If one core shows near-100% softirq while others are low, you have a single-core bottleneck. The fix is RSS or IRQ affinity tuning, not buffer sizing.

3. Check socket buffer configuration. Run sysctl net.core.rmem_max net.core.rmem_default. If rmem_max is at or near 212992, the kernel socket buffer is too small for sustained high-pps DNS traffic.

4. Check IRQ distribution. Run cat /proc/interrupts | grep eth0 (substituting your interface name). If all RX queues map to CPU 0, RSS is either not configured or the NIC does not support multiple queues. Each interrupt line shows which CPUs handle it.

5. Check NIC queue count. Run ethtool -l eth0. If the NIC reports a single combined queue, all traffic funnels through one interrupt line and one CPU core. Multi-queue NICs allow RSS to spread interrupts across cores.

6. Verify SO_REUSEPORT. Run ss -unlp | grep named and count the UDP sockets bound to port 53. With SO_REUSEPORT enabled (the default in BIND 9.16+ on Linux), BIND creates one UDP socket per worker thread, allowing the kernel to load-balance incoming packets across them. If you see only one socket, SO_REUSEPORT may be disabled.

7. Check per-thread CPU in BIND. Run pidstat -t -p $(pidof named) 1 5. If one thread is saturated while others are idle, the issue may be query processing imbalance or a single expensive code path such as DNSSEC validation or RPZ evaluation.

Metrics and signals to monitor

Fixes

Raise socket buffer sizes

If UdpRcvbufErrors is climbing and the socket buffer is at the default, raise both the max and default. These changes take effect immediately.

# Check current values
sysctl net.core.rmem_max net.core.rmem_default

# Set new values (example: 16 MiB max, 4 MiB default)
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.rmem_default=4194304

Persist these in /etc/sysctl.d/ or your distribution’s equivalent. rmem_default is the value BIND uses for its UDP sockets; rmem_max is the kernel ceiling.

Bigger buffers absorb bursts but do not fix a sustained rate mismatch. If the arrival rate exceeds BIND’s drain rate by 10,000 packets/sec, a 25 MiB buffer delays overflow by roughly 1.8 seconds (at 1.4 KB per datagram). A larger buffer only postpones the problem.

Configure RSS and IRQ affinity

For multi-queue NICs, enable RSS to distribute receive interrupts across CPU cores. Note that ethtool -L can cause momentary packet loss on some NIC drivers; test outside peak traffic if possible.

# Check current queue configuration
ethtool -l eth0

# Set N receive queues (example: 8)
# WARNING: can cause brief traffic disruption on some drivers
ethtool -L eth0 combined 8

# Verify interrupt distribution changed
cat /proc/interrupts | grep eth0

On some systems, you also need to set IRQ affinity manually by writing CPU masks to /proc/irq/<IRQ>/smp_affinity. Vendor-supplied scripts (for Intel, Mellanox, and similar NICs) automate this mapping.

Enable RPS when the NIC has fewer queues than CPUs

If the NIC cannot provide enough hardware queues, RPS (Receive Packet Steering) distributes softirq processing across additional cores in software:

# Enable RPS on a receive queue (example: all 8 CPUs, bitmask 0xFF)
echo ff > /sys/class/net/eth0/queues/rx-0/rps_cpus

RPS adds a small per-packet overhead but prevents a single core from becoming the bottleneck.

Verify SO_REUSEPORT

SO_REUSEPORT is enabled by default in BIND 9.16+ on Linux. If it has been explicitly disabled (reuseport no; in named.conf), re-enabling it allows the kernel to spread incoming UDP packets across multiple listener sockets.

One caveat: SO_REUSEPORT uses a 4-tuple hash (source IP, source port, destination IP, destination port) to select a socket. If traffic arrives from a small number of source IPs and ports (for example, all traffic proxied through one anycast router), the hash may not distribute evenly. In that case, disabling SO_REUSEPORT and relying on a single socket with a large buffer may provide more consistent per-query latency at the cost of lower aggregate throughput.

Consider compile-time tuning

If building BIND from source, the --with-tuning=large build option increases internal buffer sizes and worker thread counts beyond the defaults. These are compile-time settings that cannot be changed at runtime. Packages from most Linux distributions are not built with this option.

Prevention

• Monitor UdpRcvbufErrors continuously. Any non-zero rate during production traffic indicates packet loss invisible to BIND.
• Track per-core softirq. Alert when any single core exceeds 80% softirq sustained. Aggregate CPU average hides single-core saturation.
• Validate buffer and RSS configuration after provisioning. Test new servers under realistic packet load before going live. Kernel defaults are inadequate for high-pps DNS.
• Review anycast node health individually. Per-node saturation hides behind stable global aggregates. Monitor each node’s kernel counters separately.
• Document kernel tuning per DNS server. Socket buffer size, RSS queue count, and IRQ affinity should be in the baseline configuration, not discovered during an incident.

How Netdata helps

• Netdata collects /proc/net/snmp UDP counters including UdpRcvbufErrors at per-second resolution, so you can see the exact second drops begin rather than a smoothed average.
• Per-CPU utilization charts break out user, system, and softirq components for each core. A single core pinned at 100% softirq is immediately visible alongside the kernel drop counter.
• Interrupt distribution charts show which CPU cores handle NIC RX queues, making RSS misconfiguration obvious.
• Correlating UDP drops with per-core softirq saturation and BIND’s own query rate (collected via the BIND statistics channel) distinguishes a kernel-level packet processing bottleneck from an application-level consumption problem.

Related guides

• BIND DNSSEC validation failing: ‘broken trust chain’, ValFail, and SERVFAIL for signed domains
• BIND cache eviction storms: DeleteLRU, an undersized max-cache-size, and the pressure spiral
• BIND cache hit ratio dropping: the leading edge of recursive pain
• BIND clients-per-query and max-clients-per-query: duplicate recursion for popular names
• BIND cold cache after restart: the warming storm and elevated upstream load
• BIND DNSSEC failing from clock drift: NTP, RRSIG inception/expiry windows, and SERVFAIL
• BIND dnssec-validation disabled: the security regression that ‘fixes’ SERVFAIL
• BIND dynamic update failures: UpdateFail, denied updates, and TSIG drift
• BIND forwarding loops: recursion that never terminates and burns recursive slots
• How BIND actually works in production: a mental model for operators
• BIND inline signing silently failed: missing keys and a zone served unsigned
• BIND journal (.jnl) corruption: dynamic-update and IXFR failures that block zone load