BIND upstream query timeouts: QueryTimeout, retries, and the 30-second resolution stall

When a BIND resolver sends a recursive query upstream and gets no response, it waits. The default wait is 10 seconds, and BIND may retry the query up to 3 times before giving up. A single failed resolution can occupy a recursive-client slot for 30 seconds or more. If the upstream failure is broad enough, those slots fill, the recursive-clients limit is reached, and every new recursive query starts returning SERVFAIL.

QueryTimeout is the earliest signal that upstream responses are being lost. It appears well before RecursClients saturates or SERVFAIL spikes. This article covers how to read the counter, how the retry mechanism produces the 30-second stall, how to distinguish a single bad upstream from a systemic problem, and how to tune fail-fast behavior without making things worse.

What this means

QueryTimeout is a per-view resolver statistic, not an nsstats counter. It counts the number of outbound recursive queries that received no response before the resolver-query-timeout expired. The counter name is exactly QueryTimeout (not QryTimeout), and it lives under the resolver stats section of each view in the statistics channel JSON output.

Each timeout is expensive in two ways:

1. Slot occupation: the recursive-client slot remains allocated for the entire timeout duration plus any retries. With the default resolver-query-timeout of 10000 ms and up to 3 retries, a single unresolved query can hold a slot for 30+ seconds.

2. Retry amplification: each retry generates additional outbound traffic to the upstream. The Retry counter tracks these additional attempts. If multiple clients are querying the same name simultaneously, clients-per-query deduplication helps, but failed queries do not trigger increased deduplication. This is natural protection against runaway resource consumption, but it also means the resolver does not adapt to upstream failure by batching harder.

The compounding effect is what makes upstream timeouts dangerous. With recursive-clients at the default 1000, a sustained rate of roughly 33 stalled queries per second (1000 slots / 30 seconds per stall) exhausts all slots within 30 seconds. Once the hard limit is reached, every new recursive query returns SERVFAIL, regardless of which upstream it targets.

flowchart TD
    A[Upstream NS slow or unreachable] --> B[Outbound query: no response]
    B --> C[First timeout: QueryTimeout increments]
    C --> D[Retry sent: Retry counter increments]
    D --> E{Response received?}
    E -->|Yes| F[Slot released after delay]
    E -->|No, retries remain| C
    E -->|No, retries exhausted| G[Slot held for 30+ seconds]
    G --> H[SERVFAIL returned to client]
    G --> I[Recursive-client slot consumed]
    I --> J{Slots available?}
    J -->|Yes| K[New queries still served]
    J -->|No, limit reached| L[SERVFAIL for ALL new recursive queries]

Common causes

Quick checks

These commands are read-only and safe to run during an incident.

# QueryTimeout and Retry counters per view
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(f'{v}: {k}: {s}') for v,vd in d.get('views',{}).items() \
  for k,s in sorted(vd.get('resolver',{}).get('stats',{}).items()) \
  if k in ('QueryTimeout','Lame','QuerySockFail','QueryAbort','Retry','OtherError')]"

# Current recursive client utilization (gauge value, not a counter)
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  print('RecursClients:', d.get('nsstats',{}).get('RecursClients', 'N/A'))"

# Which upstream nameservers are in-flight queries waiting on
rndc recursing | awk '{print $NF}' | sort | uniq -c | sort -rn | head -10

# RTT distribution per view (shift to high buckets = upstream degradation)
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(f'{v}: {k}: {s}') for v,vd in d.get('views',{}).items() \
  for k,s in sorted(vd.get('resolver',{}).get('stats',{}).items()) if 'RTT' in k]"

# Test a specific upstream directly from the resolver host
dig +time=2 +tries=1 @<upstream-ip> example.com A

# Check resolver-query-timeout value in running config
named-checkconf -p /etc/named.conf | grep resolver-query-timeout

The statistics channel port (8653 above) varies by configuration. Common values are 8053 and 8653. Check the statistics-channels block in your named.conf.

How to diagnose it

1. Confirm QueryTimeout is elevated above baseline. Pull the per-view resolver stats and express QueryTimeout as a percentage of total outbound queries. Normal is under 2%. Above 5% warrants investigation. Above 20% indicates widespread upstream reachability failure. These are cumulative counters since process start: you need two samples to compute a rate.

2. Check if RecursClients is climbing. If QueryTimeout is high and RecursClients is trending toward the recursive-clients limit (default 1000), you are in the cascade. Prioritize identifying and isolating the bad upstream before the limit is reached.

3. Identify the offending upstream. Run rndc recursing and look for concentration. If most in-flight queries target the same upstream IP or a small set of IPs, the problem is narrow. If queries are spread across many upstreams, the problem is systemic (network path, source port exhaustion, or local resource limit).

4. Test the upstream directly. From the resolver host, query the suspected upstream nameserver with a short timeout: dig +time=2 +tries=1 @<upstream-ip> example.com A. If this also times out, the upstream or the network path is the problem. If it succeeds, the issue may be intermittent rate-limiting or load-dependent behavior at higher volumes.

5. Check for version-specific bugs. BIND 9.18.8 through 9.18.10 had a bug where the resolver retried the same unresponsive authoritative nameserver three times before trying the next one, instead of moving on after the first timeout. Root cause was rctx_timedout() setting rctx->resend = true instead of rctx->next_server = true. Fixed in 9.16.37, 9.18.11, and 9.19.9 (January 2023 releases). If you are on an affected version and see excessive retries against a single upstream, upgrading resolves the behavior.

6. Check for CVE-2026-5950. This vulnerability causes an unbounded resend loop in the BIND resolver during bad-server handling. A remote attacker can trigger it to cause severe resource exhaustion. It affects BIND 9.18.36 through 9.18.48, 9.20.8 through 9.20.22, and 9.21.7 through 9.21.21. Fixed in 9.18.49, 9.20.23, and 9.21.22. If QueryTimeout and Retry counters are climbing uncontrollably and you are on an affected version, this is a likely cause.

7. Differentiate from other resolver failure modes. High QueryTimeout with normal RTT buckets suggests packet loss on the outbound path. High QueryTimeout with high RTT buckets (1600+) suggests upstream slowness. The Lame counter increasing alongside QueryTimeout suggests broken delegation (the delegated server is not authoritative for the zone). QuerySockFail increasing suggests local socket errors or source port exhaustion.

Metrics and signals to monitor

Fixes

Identify and isolate the bad upstream

If rndc recursing shows queries concentrated on one or two upstream IPs, the upstream is the problem. Your options depend on your deployment:

• Forwarder configuration: if you are using forwarders, verify each forwarder is reachable. Remove or replace a dead forwarder and apply: rndc reconfig.
• Recursive resolution: you cannot remove an authoritative nameserver from the delegation, but you can flush cached entries for the affected domain to clear stuck state. Use rndc flushname <domain> for a targeted flush. rndc flush without arguments clears the entire cache, which forces cache warming and increases outbound query load. Use with caution on busy resolvers.
• Rate-limited upstream: if the upstream is rate-limiting your resolver (common with some public DNS providers under high load), reducing clients-per-query will not help. Consider adding forwarders or secondary resolvers to distribute load.

Tune resolver-query-timeout for fail-fast behavior

The resolver-query-timeout option controls the maximum time BIND spends on a single recursive resolution attempt. Lowering it reduces the slot occupation per failed query, freeing recursive-client slots faster.

# In named.conf, within options or a view block:
# resolver-query-timeout 5000;
# Then: rndc reconfig

Value interpretation trap: BIND treats values 1 through 300 as seconds, and values 301 and above as milliseconds. So resolver-query-timeout 30; means 30 seconds, not 30 milliseconds. This is a common misconfiguration that makes timeouts worse instead of better. Setting to 0 uses the default (10000 ms).

ISC does not recommend reducing resolver-query-timeout below 10 seconds in most operational environments. Lower values cause higher SERVFAIL rates because complex delegation chains and DNSSEC validation may require multiple round trips that exceed a tight timeout. Use this tuning only as a temporary measure during an active incident where the cascade is already causing widespread SERVFAIL.

Raise recursive-clients to absorb the stall

If the upstream problem is temporary and you need headroom while it resolves:

# In named.conf options block:
# recursive-clients 2000;
# Then: rndc reconfig

Raising recursive-clients gives the resolver more slots to absorb stalled queries, but each slot consumes a file descriptor and memory. If your FD limit is at or near the default 1024, raising recursive-clients without also raising the FD limit will cause FD exhaustion. Check current FD usage first:

# Current FD count and limit
ls /proc/$(pgrep -x named)/fd | wc -l
grep "Max open files" /proc/$(pgrep -x named)/limits

Address the root cause

Temporary measures only buy time. The actual fix depends on the root cause:

• Upstream nameserver outage: contact the operator or wait for recovery. The resolver’s RTT-based server selection will deprioritize slow upstreams automatically, but only if alternatives exist in the delegation.
• Network path failure: work with the network team to identify the black hole or firewall rule. Stateful firewalls that inspect DNS UDP can create bottlenecks.
• Source port exhaustion: widen the ephemeral port range (sysctl net.ipv4.ip_local_port_range) or reduce concurrent query volume.
• Version-specific bug or CVE: upgrade BIND to a fixed release. This is the only permanent fix for retry-path bugs.

Prevention

• Monitor QueryTimeout as a percentage of outbound queries, per view. A sustained rate above 2% is the earliest actionable signal that upstream reachability is degrading. Express as a ratio, not an absolute count.
• Track RecursClients against the recursive-clients limit. Daily peak should not exceed 50% of the limit. This provides headroom for upstream slowdowns and attack absorption. The degradation curve is cliff-edge: once you approach the limit, the resolver goes from stressed to broken quickly.
• Periodically sample rndc recursing to identify which upstream nameservers your resolver depends on most heavily. A sudden change in the top upstreams can indicate a delegation change or a new failure pattern.
• Keep BIND patched. The 9.18.8-9.18.10 retry bug and CVE-2026-5950 both caused excessive retries that mimic upstream failure. Running current stable releases avoids known retry-path bugs.
• Verify resolver-query-timeout is set intentionally. The value interpretation trap (1-300 as seconds, 301+ as milliseconds) has caused operators to accidentally set 30-second timeouts when they meant 30 milliseconds, or the reverse.

How Netdata helps

Netdata’s BIND collector surfaces the resolver statistics that matter for this failure mode at per-second resolution:

• QueryTimeout per view: the earliest signal that upstream responses are being lost. Express as a percentage of outbound queries. Per-second collection lets you pinpoint the exact moment the rate changes.
• Retry per view: correlates with QueryTimeout to show whether retries are recovering or compounding the stall. If both rise together, retries are not succeeding.
• RecursClients as a gauge: the saturation indicator. When QueryTimeout rises and RecursClients climbs toward the limit, the cascade is active.
• QryRTT bucket distribution per view: shifts toward higher RTT buckets can precede timeouts, giving early warning before QueryTimeout increments.
• QrySERVFAIL rate: the user-visible effect. When recursive-clients is exhausted, SERVFAIL spikes across many unrelated domains. Correlating SERVFAIL with QueryTimeout and RecursClients confirms the upstream cascade as the cause rather than a zone-specific or DNSSEC problem.

Related guides

• BIND forwarding loops: recursion that never terminates and burns recursive slots
• How BIND actually works in production: a mental model for operators
• BIND monitoring checklist: the signals every production resolver and authoritative server needs
• BIND monitoring maturity model: from survival to expert
• BIND ’no more recursive clients: quota reached’: the recursive-clients circuit breaker
• named not responding on port 53: total outage versus UDP-works-TCP-fails
• BIND resolver NumFetch per view: per-view recursive pressure in split-horizon setups
• BIND NXDOMAIN spike: DGA malware, water torture, and Windows suffix search lists
• BIND recursive resolution cascade: one slow upstream taking down all resolution
• BIND RecursClients climbing toward the limit: reading the recursive saturation gauge
• BIND REFUSED responses: ACL denials, recursion policy, and clients that get locked out
• rndc not responding: control-plane failure while queries still work