BIND recursive resolution cascade: one slow upstream taking down all resolution

SERVFAIL responses are flooding your recursive resolver. Users cannot resolve dozens of unrelated domains. But named is running, CPU looks moderate, and the authoritative zones on the same instance are still answering fine. This is the recursive resolution cascade: a single slow or unreachable upstream authoritative server consumes all available recursive-clients slots, and BIND returns SERVFAIL for queries that have nothing to do with the failing upstream.

Each in-flight recursive query targeting a slow upstream holds its slot for the full resolver-query-timeout duration (default 10 seconds). With retries across multiple nameservers, a single failed resolution can occupy a slot for 30 seconds or more. As stuck slots accumulate, capacity shrinks for unrelated queries. Once the hard limit is reached, every new recursive query receives SERVFAIL immediately. The signals that point to the real problem are in resolver internals that most teams do not monitor proactively.

What this means

BIND caps concurrent recursive queries using the recursive-clients option (default 1000). Each in-flight recursive query occupies one slot until it completes or times out. A soft quota kicks in at 90% of the limit (default 900), at which point BIND starts aborting the oldest queries to free slots. At the hard limit (1000), all new recursive queries receive SERVFAIL immediately.

When an upstream authoritative server becomes slow or unreachable, queries routed to it hold their slots for the duration of resolver-query-timeout. If enough queries target the same slow upstream in parallel, slots accumulate faster than they drain, and the resolver runs out of capacity for entirely unrelated queries.

flowchart TD
    A["Incoming queries"] --> B{"Cache hit?"}
    B -->|"Yes"| C["Answer from cache
no slot needed"]
    B -->|"Cache miss"| D["Allocate recursive slot"]
    D --> E{"Upstream condition"}
    E -->|"Healthy server"| F["Fast response
slot freed"]
    E -->|"Slow or dead server"| G["Slot held for
timeout ~10s plus retries"]
    G --> H["Stuck slots accumulate"]
    H --> I{"recursive-clients
utilization"}
    I -->|"Above 90%"| J["Soft quota:
abort oldest queries"]
    I -->|"At 100%"| K["Hard limit:
all new recursive
queries get SERVFAIL"]

SERVFAIL from this pattern is broad: it affects many unrelated domains, not just domains served by the problematic upstream. Authoritative zones on the same BIND instance continue working because they are served from local zone data and do not consume recursive slots. CPU often stays moderate because worker threads are blocked on network responses, not computing.

Common causes

Quick checks

Run these read-only commands to confirm the cascade pattern and identify the offending upstream. Adjust the port and authentication to match your statistics-channels configuration.

# Check recursive client count against the limit (default 1000)
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  print('RecursClients:', d.get('nsstats',{}).get('RecursClients', 'N/A'))"

# Check SERVFAIL rate
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  print('QrySERVFAIL:', d.get('nsstats',{}).get('QrySERVFAIL','N/A'))"

# Identify which upstream nameservers queries are piling up against.
# Output format varies by BIND version; adjust parsing to match your output.
rndc recursing | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+#[0-9]+' | sort | uniq -c | sort -rn | head -10

# Check resolver timeouts (per-view stat)
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(f'{v}: QueryTimeout={vd.get(\"resolver\",{}).get(\"stats\",{}).get(\"QueryTimeout\",\"N/A\")}') \
  for v,vd in d.get('views',{}).items()]"

# Check upstream RTT distribution (high RTT buckets indicate upstream slowness)
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(f'{v}: {k}: {s}') for v,vd in d.get('views',{}).items() \
  for k,s in sorted(vd.get('resolver',{}).get('stats',{}).items()) if 'RTT' in k or 'rtt' in k]"

# Verify authoritative zones still work (they should if this is a cascade)
dig +time=2 +tries=1 +norecurse @127.0.0.1 <your-zone> SOA

# Confirm CPU is moderate (threads blocked, not computing)
pidstat -p $(pgrep -x named) 1 3

How to diagnose it

1. Confirm the cascade pattern. Broad SERVFAIL across unrelated domains, with RecursClients approaching or at the limit, and authoritative zones still resolving. If SERVFAIL is limited to specific zones, see the SERVFAIL tracing guide.

2. Identify the offending upstream. The rndc recursing output shows every in-flight recursive query and the upstream it is waiting on. Aggregate by upstream IP to find the pile-up source. If most in-flight queries target the same nameserver or a small set of nameservers, you have found the bottleneck.

3. Check QueryTimeout rate. Elevated timeouts in per-view resolver stats confirm upstreams are not responding. The correct counter is QueryTimeout, a per-view resolver stat, not an nsstats counter. Compare the timeout rate against your baseline.

4. Examine RTT distribution. A shift toward higher RTT buckets (especially the 1600+ bucket) indicates upstream degradation. If only specific upstreams show high RTT while others are normal, the problem is localized.

5. Rule out DNSSEC. Run dig @127.0.0.1 <domain> +cd on a failing domain. If it works with checking disabled but not without, DNSSEC validation is the issue, not upstream slowness. See the SERVFAIL guide for the DNSSEC time bomb pattern.

6. Rule out water torture. Check NXDOMAIN rate and query name cardinality. If most failing queries are random subdomains of one parent domain, you are under a random subdomain attack, not a cascade. See the NXDOMAIN spike guide.

7. Check for forwarding loops. If your BIND is in forwarder mode and the forwarder is slow or unreachable, the cascade behavior is identical. Verify forwarder health independently.

Metrics and signals to monitor

Fixes

Identify and flush stuck entries

The first action during an active cascade is to identify the offending upstream and flush any stuck cache entries for the affected domains.

# Flush cache for a specific domain (targeted, limited impact)
rndc flushname <domain>

# WARNING: Flushes the entire cache. On a busy resolver this causes
# a massive spike in upstream queries as the cache rebuilds, which
# can worsen the cascade. Use only as a last resort.
rndc flush

Flushing stuck entries frees slots immediately but does not prevent new queries from piling up again if the upstream is still slow. Prefer rndc flushname <domain> for targeted cleanup over rndc flush.

Lower resolver-query-timeout to fail fast

Reducing resolver-query-timeout makes BIND give up on slow upstreams sooner, freeing slots faster. The default is 10000 ms (10 seconds). In BIND 9.20.15 and later, the minimum was lowered to 301 ms, giving operators the option to fail fast in targeted scenarios.

// In named.conf, within options or a view:
resolver-query-timeout 5000;  // 5 seconds: fail faster on slow upstreams

Lower values cause a higher rate of SERVFAIL responses because BIND has less time to progress through complex delegation chains and cache-loading. Use this as a temporary measure during an active cascade, not a permanent setting for general operation.

On BIND versions earlier than 9.20.15, the minimum is 10000 ms, so this tuning option is unavailable.

Remove or replace problematic forwarders

If your resolver uses forwarders and one is slow or unreachable, remove it from the forwarders list and reload the configuration. The cascade behavior with forwarders is identical to direct recursion: each query to a slow forwarder holds a slot for the full timeout.

# Validate configuration before reloading
named-checkconf /etc/named.conf

# Reload configuration without restarting the daemon
rndc reconfig

Enable serve-stale as a buffer

Serve-stale allows BIND to answer from expired cache entries while retrying the upstream, reducing the number of queries that need fresh recursive slots. This requires stale-answer-enable yes in your configuration.

// In named.conf options or view:
stale-answer-enable yes;
stale-refresh-time 30;  // default 30 seconds

The version interaction matters. Prior to BIND 9.16.9 and 9.17.7, BIND waited for the full resolver-query-timeout before serving a stale answer, which did not help during a cascade. After this change, BIND serves the stale answer immediately if a refresh attempt has previously failed and continues serving it for stale-refresh-time seconds. This makes serve-stale effective as a cascade buffer on modern versions.

Be aware of CVE-2023-2911: if your BIND is configured with both stale-answer-enable yes and stale-answer-client-timeout 0, a quota exhaustion scenario can cause a stack overflow crash. Fixed in 9.16.42 and 9.18.16. Workaround: set stale-answer-client-timeout to off or a non-zero value.

Increase recursive-clients (capacity, not cure)

Raising recursive-clients above the default 1000 gives more headroom but does not fix the root cause. Each recursive client uses approximately 20 KB of memory and consumes file descriptors. Setting it too high without sufficient FDs and memory causes other resource exhaustion.

// In named.conf options:
recursive-clients 2000;

This buys time during a cascade but masks the underlying upstream problem. Monitor FD usage after any increase. The files option for FD limits is deprecated in BIND 9.18 and removed in 9.20. Use OS-level limits such as ulimit or systemd LimitNOFILE.

Prevention

• Monitor RecursClients as a percentage of the limit. Track daily peak utilization. Daily peak should not exceed 50% of recursive-clients to provide headroom for traffic spikes, upstream slowdowns, and attack absorption. See the monitoring checklist for the full signal set.

• Track upstream RTT distribution. A gradual shift toward higher RTT buckets is an early warning that an upstream is degrading, before it becomes slow enough to cause a cascade.

• Configure serve-stale proactively. Do not wait for an incident to enable it. Serve-stale turns hard failures into degraded-but-functional responses during upstream outages.

• Keep BIND patched. CVE-2026-5950 (unbounded recursion loop against bad servers, fixed in 9.21.22) and CVE-2026-3593 (heap use-after-free in DNS-over-HTTPS, affecting 9.20.0 through 9.20.22 and 9.21.0 through 9.21.21) both directly affect cascade behavior.

• Diversify upstream dependencies. If you forward to a single upstream resolver, a failure there cascades directly. Multiple independent upstream paths provide redundancy.

• Sample rndc recursing periodically. During incidents this command is invaluable, but it is almost never collected proactively. Periodic sampling establishes a baseline of which upstreams your resolver depends on and their normal response patterns.

How Netdata helps

• Per-second RecursClients gauge shows slot pressure building in real time, not after SERVFAIL has already started. Per-second granularity catches fast cascades that 60-second polling misses.
• SERVFAIL rate correlation against recursive client count directly identifies the cascade pattern: both rising together signals upstream-induced slot exhaustion, while SERVFAIL rising alone points to DNSSEC or zone issues.
• Per-view resolver stats (QueryTimeout, RTT distribution, NumFetch) show which view and which upstream is degrading, narrowing the diagnosis from “DNS is broken” to a specific upstream path.
• Cache hit ratio trending reveals the self-reinforcing loop: as the cascade blocks cache population, hit ratio falls, which increases outbound queries, which increases slot pressure.
• Anomaly detection on RecursClients catches gradual slot pressure growth that fixed thresholds miss.

For the full monitoring signal taxonomy across BIND roles, see the monitoring maturity model.

Related guides

• How BIND actually works in production: a mental model for operators
• BIND monitoring checklist: the signals every production resolver and authoritative server needs
• BIND monitoring maturity model: from survival to expert
• named not responding on port 53: total outage versus UDP-works-TCP-fails
• BIND NXDOMAIN spike: DGA malware, water torture, and Windows suffix search lists
• BIND REFUSED responses: ACL denials, recursion policy, and clients that get locked out
• rndc not responding: control-plane failure while queries still work
• BIND SERVFAIL responses: what a DNS SERVFAIL actually means and how to trace the cause