BIND CPU saturation: single-core bottlenecks, DNSSEC crypto, and per-thread contention

BIND CPU saturation often hides from aggregate monitoring. Because named runs one worker thread per CPU core through netmgr, aggregate process CPU can read 25% on a 4-core host while a single core is pinned at 100%. Standard monitoring that checks total CPU utilization sees a healthy daemon. Clients see latency spikes and intermittent timeouts.

The core diagnostic challenge: BIND’s statistics channel reports no per-thread CPU data. You must measure at the OS level with mpstat, pidstat, or direct /proc inspection. Without per-core visibility, the symptom presents as unexplained latency with no obvious cause.

Primary CPU consumers: DNSSEC cryptographic operations (validation on recursive resolvers, inline signing on authoritative servers), large zone loads at startup or reload, malformed-packet parsing under attack, RPZ rule evaluation, and query mixes that concentrate expensive work on specific threads. RSA signatures larger than 2048 bits are disproportionately costly per query compared to ECDSA.

What single-core saturation means

One netmgr worker thread has become the bottleneck for the entire daemon. netmgr distributes incoming queries among worker threads, but distribution is not perfectly load-balanced for all workload shapes. When one thread is saturated by DNSSEC validation or packet parsing, queries assigned to it queue behind each other. Other worker threads may sit idle.

The degradation is gradual, not cliff-edge. Latency increases proportionally to queue depth on the saturated core. It does not produce SERVFAIL immediately. Eventually, client-side timeouts (typically 2-5 seconds for stub resolvers) start firing, and queries fail. By that point, the thread has been saturated for minutes or hours.

Softirq pressure compounds the problem. On Linux, network interrupt handling runs in softirq context on the CPU that receives the hardware interrupt. If NIC IRQ affinity concentrates all packet interrupts on one core, and that same core also runs a netmgr worker thread, the worker competes with softirq for CPU cycles. The thread never gets enough time to drain the socket buffer.

flowchart TD
    A["Latency or timeouts, CPU looks normal"] --> B["Check per-core: mpstat -P ALL"]
    B --> C{"One core near 100%?"}
    C -- "Yes" --> D{"Softirq dominant?"}
    D -- "Yes" --> E["IRQ affinity issue: check /proc/interrupts"]
    D -- "No" --> F["Worker thread contention: pidstat -t"]
    F --> G["Check: Val counters, QTypes, RPZ, zone loads"]
    C -- "No" --> H["Check kernel UDP drops and socket buffers"]

Common causes

Quick checks

All read-only and safe during production traffic.

# Per-core CPU utilization: look for one core near 100% while others are low
mpstat -P ALL 1 5

# Per-thread CPU for named: identifies which worker thread is saturated
pidstat -t -p $(pgrep -x named) 1 5

# Kernel UDP receive buffer errors: packets dropped before BIND sees them
cat /proc/net/snmp | grep Udp

# Interrupt distribution for your NIC (replace eth0 with your interface name)
cat /proc/interrupts | grep -i eth0

# DNSSEC validation activity in BIND statistics
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(f'{v}: {k}={s}') for v,vd in d.get('views',{}).items() \
  for k,s in vd.get('resolver',{}).get('stats',{}).items() if k.startswith('Val')]"

# Query type distribution for anomalous patterns
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  print(dict(sorted(d.get('qtypes',{}).items(), key=lambda x:-x[1])[:10]))"

# Functional latency test
dig @127.0.0.1 example.com A +time=2 +tries=1 | grep "Query time"

The statistics channel port (8653 above) depends on your configuration. Common values are 8053 and 8653.

How to diagnose

1. Confirm single-core saturation. Run mpstat -P ALL 1 5. Look for one CPU column consistently near 100% while others are moderate. This is the signature.

2. Separate softirq from user-space. In the same mpstat output, check the %soft column. If softirq is high on the same core that is saturated, the bottleneck is network interrupt processing, not BIND application logic. The fix path diverges here: IRQ affinity tuning versus workload reduction.

3. Identify the saturated worker thread. Run pidstat -t -p $(pgrep -x named) 1 5. Look for one thread with disproportionately high CPU. The thread TID does not directly tell you which queries it is processing, but it confirms thread-level contention, not system-wide load.

4. Correlate with BIND signals. Pull DNSSEC validation counters (ValAttempt, ValOk, ValFail), query type distribution (qtypes), and RPZ rewrite rate from the statistics channel. If ValAttempt is high and the resolver is processing many signed domains, DNSSEC crypto is the likely consumer. If QType distribution shows anomalous ANY or TXT spikes, an attack may be driving expensive parsing.

5. Check kernel UDP drops. Run cat /proc/net/snmp | grep Udp and examine RcvbufErrors. Non-zero values mean the kernel is dropping packets before BIND reads them. This is invisible to BIND’s own counters and often coexists with single-core saturation when softirq is the bottleneck.

6. Check BIND version against known CPU-exhaustion CVEs. KeyTrap (CVE-2023-50387) and CVE-2023-50868 allow a single crafted DNSSEC response to cause prolonged CPU consumption on validating resolvers. Fixed in BIND 9.16.47, 9.18.23, and 9.19.21 for CVE-2023-50387. If running an older version on a recursive resolver with DNSSEC validation enabled, this is a candidate for unexplained CPU spikes from specific query patterns.

Metrics and signals to monitor

Fixes

IRQ affinity imbalance

If mpstat shows high %soft on one core, network interrupts are not distributed across CPUs. Check /proc/interrupts to see which core handles your NIC’s RX queue.

• Disable or reconfigure irqbalance. If it is not distributing effectively across cores for your NIC, stop it and set affinity manually.
• Set smp_affinity manually. Write a CPU bitmask to /proc/irq/<irq_number>/smp_affinity for the NIC’s interrupt line. This takes effect immediately.
• Use multi-queue NICs. Ensure each RX queue maps to a different CPU.

This is the highest-leverage fix when softirq is the bottleneck. BIND cannot solve it internally; the kernel controls interrupt routing.

DNSSEC validation load

On recursive resolvers, DNSSEC validation is a primary CPU consumer. Specific conditions spike it: cache miss storms across many unique signed domains, or queries to zones signed with RSA keys larger than 2048 bits.

Remediation:

• Ensure cache hit ratio is healthy. A falling hit ratio means more cache misses, which means more DNSSEC validation per query.
• Verify BIND version patches KeyTrap. CVE-2023-50387 allows a single response to cause disproportionate CPU consumption on unpatched versions.
• Do not disable DNSSEC validation. Setting dnssec-validation no makes the resolver accept forged responses. It suppresses the CPU symptom by removing security entirely.

DNSSEC signing load (authoritative)

For authoritative servers with inline signing, RSA signatures are significantly more expensive to compute than ECDSA. If you control the zone’s signing policy, consider ECDSA (P-256) for equivalent security with lower computational cost per signature.

Check signing freshness with rndc signing -list <zone>. If signing is stuck or failing silently, signatures may be approaching expiry. Missing .signed.jnl indicates the zone is not being signed at all.

Worker thread count

By default, named creates one worker thread per detected CPU core. The -n flag overrides the number of UDP listener threads. On systems where BIND shares CPUs with other services, reducing -n can reduce contention by giving each thread a more predictable CPU budget.

Do not increase -n beyond the physical core count. Additional threads compete for CPU and can worsen single-core saturation through context switching.

Socket buffer sizing

If UdpRcvbufErrors is non-zero, the kernel UDP receive buffer is too small for the packet rate. Increase it:

# Check current values
sysctl net.core.rmem_max net.core.rmem_default

# Increase (takes effect immediately but is NOT persistent across reboots)
sysctl -w net.core.rmem_max=4194304

Persist changes in /etc/sysctl.d/. BIND’s so-rcvbuf option in named.conf can also set the SO_RCVBUF socket option per listener.

Query logging

If query logging (querylog category) is enabled in production, disable it. At high QPS, log I/O competes with query processing for CPU and disk. Use the statistics channel for aggregate counters instead, and enable query logging only temporarily during investigations.

Version upgrade

If running BIND below 9.18.23 (or 9.16.47 on the legacy branch), upgrade to address KeyTrap (CVE-2023-50387) and CVE-2023-50868. A single crafted DNSSEC response can burn CPU for seconds on unpatched resolvers.

Prevention

• Monitor per-core CPU, not just aggregate. Per-core charts at per-second resolution reveal the one-core-pinned pattern immediately.
• Monitor UdpRcvbufErrors continuously. Any non-zero rate during production hours indicates packet loss that BIND cannot see or report.
• Track DNSSEC validation counters as a trend. Rising ValAttempt rate at constant query volume means more signed domains being resolved and more CPU per query.
• Run named-checkconf before every reload. Configuration errors that cause zone load failures waste CPU on failed load attempts and retry cycles.
• Keep BIND patched. DNSSEC-related CPU exhaustion CVEs have been a recurring pattern.
• Review RPZ dataset sizes periodically. Growth in RPZ rules directly increases CPU cost per query.

How Netdata helps

• Per-core CPU utilization at per-second resolution reveals the one-core-pinned pattern that aggregate averages hide.
• Softirq time per core distinguishes IRQ affinity imbalance from application-level contention.
• UDP receive buffer errors collected from /proc/net/snmp surface kernel-level packet drops invisible to BIND’s own counters.
• DNSSEC validation counters (ValAttempt, ValOk, ValFail) correlated with per-core CPU confirm whether crypto load is driving the saturated thread.
• Anomaly detection on per-core CPU and query rate catches gradual degradation before it crosses a static threshold.

Related guides

• BIND DNSSEC validation failing: ‘broken trust chain’, ValFail, and SERVFAIL for signed domains
• BIND cache eviction storms: DeleteLRU, an undersized max-cache-size, and the pressure spiral
• BIND cache hit ratio dropping: the leading edge of recursive pain
• BIND clients-per-query and max-clients-per-query: duplicate recursion for popular names
• BIND cold cache after restart: the warming storm and elevated upstream load
• BIND DNSSEC failing from clock drift: NTP, RRSIG inception/expiry windows, and SERVFAIL
• BIND dnssec-validation disabled: the security regression that ‘fixes’ SERVFAIL
• BIND dynamic update failures: UpdateFail, denied updates, and TSIG drift
• BIND forwarding loops: recursion that never terminates and burns recursive slots
• How BIND actually works in production: a mental model for operators
• BIND inline signing silently failed: missing keys and a zone served unsigned
• BIND journal (.jnl) corruption: dynamic-update and IXFR failures that block zone load