BIND resolver RTT distribution shifting high: upstream nameserver degradation

The QryRTT histogram in BIND’s per-view resolver statistics is shifting toward higher millisecond buckets. What was once dominated by RTT10 and RTT100 (sub-100ms responses from upstream authoritative servers) is now accumulating in RTT500, RTT800, and RTT1600. In severe cases, most outbound recursive queries land in the overflow RTT1600+ bucket.

This metric is not client-perceived latency. The QryRTT counters measure the round-trip time of BIND’s outbound recursive queries to upstream authoritative nameservers. A rightward shift means the servers BIND depends on for cache misses are taking longer to respond. BIND has no native inbound client latency histogram; if you need end-to-end latency from the client perspective, use dnstap or external measurement.

A sustained RTT shift is a leading indicator. It precedes the more visible symptoms: recursive-clients slots filling, SERVFAIL rates climbing, and end-user timeouts. Catching the shift early gives you time to identify the degraded upstream and mitigate before the resolver cascades into broad failure.

What this means

BIND exposes six RTT histogram buckets per view in the resolver statistics section of the statistics channel:

Each counter accumulates the number of outbound recursive queries whose RTT fell within that range. The final bucket (1600+) is the overflow bucket for queries at or above 1600ms.

A healthy recursive resolver serving typical internet traffic shows the vast majority of outbound queries in RTT10 and RTT100, with a modest tail in RTT500. When RTT500, RTT800, or RTT1600 captures a growing share, the upstream nameservers BIND recurses to are responding more slowly.

Three properties of these counters affect interpretation:

Cumulative, not instantaneous. The counters accumulate since process start or last statistics reset. A resolver running for weeks with mostly fast upstream responses will show enormous RTT10 and RTT100 counters even if the last hour was terrible. You must compute deltas over your monitoring interval to see the current distribution rather than the all-time distribution.

Recursive resolvers only. Authoritative-only servers do not issue recursive queries and will not populate these counters meaningfully. If your BIND instance serves zones but does not have recursion yes, the RTT buckets will be empty or near-zero.

Per-view. In split-horizon setups, each view maintains its own RTT distribution. A shift in one view but not others can isolate which upstream path is degraded. Always check whether the shift is global or view-specific.

An internal BucketSize stat exists in the raw statistics output, but the Netdata BIND collector drops it. Do not rely on it for monitoring.

flowchart TD
    A["QryRTT buckets shifting high"] --> B{"QueryTimeout rising?"}
    B -->|"Yes"| C["Upstream dropping packets"]
    B -->|"No"| D["Upstream slow but alive"]
    C --> E{"RecursClients climbing?"}
    E -->|"Yes"| F["Cascade risk active"]
    E -->|"No"| G["Watch for escalation"]
    F --> H["Identify upstream: rndc recursing"]
    D --> I["Test upstream directly"]

Common causes

Quick checks

These are safe, read-only commands. Adjust the statistics channel port (8653 in these examples) to match your configuration.

# Check per-view RTT bucket distribution
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(f'{v}: {k}: {s}') for v,vd in d.get('views',{}).items() \
  for k,s in sorted(vd.get('resolver',{}).get('stats',{}).items()) if 'RTT' in k or 'rtt' in k]"

# Check resolver failure counters per view (QueryTimeout, Lame, Retry, etc.)
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(f'{v}: {k}: {s}') for v,vd in d.get('views',{}).items() \
  for k,s in sorted(vd.get('resolver',{}).get('stats',{}).items()) \
  if k in ('QueryTimeout','Lame','QuerySockFail','QueryAbort','Retry','SERVFAIL','Truncated','EDNS0Fail','OtherError')]"

# Check recursive clients (gauge, not cumulative)
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  print('RecursClients:', d.get('nsstats',{}).get('RecursClients', 'N/A'))"

# Identify which upstream nameservers are causing pile-up
# Output format varies by BIND version; adjust field extraction as needed
rndc recursing | awk '/#/{print $NF}' | sort | uniq -c | sort -rn | head -10

# Test a specific upstream nameserver directly
dig @<upstream-ip> example.com A +time=2 +tries=1 | grep "Query time"

# Check cache hit ratio per view
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(f'{v}: Hits={cs.get(\"CacheHits\",0)} Misses={cs.get(\"CacheMisses\",0)} Ratio={cs.get(\"CacheHits\",0)/(cs.get(\"CacheHits\",0)+cs.get(\"CacheMisses\",1))*100:.1f}%') \
  for v,vd in d.get('views',{}).items() if (cs:=vd.get('resolver',{}).get('cachestats',{}))]"

# Check resolver-query-timeout setting
named-checkconf -p /etc/named.conf 2>/dev/null | grep resolver-query-timeout || echo "not set (default: 10000ms)"

How to diagnose it

1. Confirm the shift is real, not a cold-start artifact. Check uptime. If the resolver restarted recently (under 30-60 minutes), the RTT distribution will be skewed because BIND is re-probing upstream server RTTs from scratch. Wait for cache and RTT estimates to stabilize.

2. Compute deltas, not raw values. The QryRTT counters are cumulative. Sample the statistics channel twice with a known interval (for example, 60 seconds apart) and compute the delta for each bucket. The delta distribution is what matters. A single snapshot of cumulative counters tells you the all-time average, which may mask a recent degradation.

3. Check whether the shift is global or per-view. If only one view shows the shift, the problem is likely in the upstream path specific to that view’s configuration (forwarders, network routing, or specific zones). A global shift across all views points to a broader upstream or network issue.

4. Correlate with QueryTimeout. This is the single most important correlation. If RTT buckets shift high and QueryTimeout is also rising, upstream nameservers are not just slow but are dropping or failing to respond to packets. If QueryTimeout is flat, the upstream is slow but still answering. The distinction determines urgency: a slow-but-alive upstream degrades performance, while a timing-out upstream risks cascade failure.

5. Check RecursClients and NumFetch. Each slow outbound query holds a recursive-client slot. If RecursClients is climbing toward the configured limit (default 1000, soft quota at 900), the resolver is approaching cascade failure. Per-view NumFetch shows which view is consuming the most slots.

6. Identify the specific upstream. Run rndc recursing and look at which upstream nameservers appear most frequently in the in-flight query list. The aggregation command in Quick checks gives a top-N of upstreams being waited on.

7. Test the identified upstream directly. Use dig @<upstream-ip> from the resolver host and from an external host. Compare RTT. If the upstream is slow from both locations, the problem is upstream-side. If it is only slow from the resolver host, suspect a local network path issue.

8. Check for EDNS0 or fragmentation issues. Elevated Truncated or EDNS0Fail counters suggest the upstream or an intermediate firewall is mishandling EDNS0 or causing fragmentation. This forces TCP fallback, which is inherently slower and consumes more file descriptors.

9. Consider version-specific bugs. Before BIND 9.18.48, when query responses timed out the resolver could incorrectly increase the regular response counters even though no response was received (GL #5193, fixed in 9.18.48). If you are running an older 9.18.x, do not over-interpret fine-grained resolver counter differences around timeouts.

Metrics and signals to monitor

Fixes

Upstream nameserver degradation

If a specific upstream nameserver is consistently slow, BIND’s RTT-based server selection algorithm will automatically deprioritize it. BIND maintains internal RTT estimates per nameserver and preferentially queries faster servers within the same delegation. This is self-healing for partial failures: a single slow authoritative server among several healthy ones may not noticeably impact resolution.

However, if all authoritative servers for a zone or TLD are slow, BIND has no alternative. In this case:

• Lower resolver-query-timeout temporarily to fail fast rather than holding recursive-client slots for the full timeout duration (default 10 seconds). The option accepts values from 301ms to 30000ms and caps the total time spent attempting one recursive query, so a slot is held for at most that duration. Lowering it reduces the blast radius of upstream failures but increases the risk of premature SERVFAIL for legitimately complex delegation chains. Use as a temporary mitigation, not a permanent setting.

• Flush stuck cache entries if negative caching is compounding the problem. rndc flushname <domain> clears entries for a specific name without flushing the entire cache. A full rndc flush is more disruptive and should be reserved for severe cases.

Network path issues

If the upstream is slow only from the resolver host, investigate the local network path:

• Check for packet loss with mtr or traceroute to the upstream IP.
• Verify no firewall or NAT is constraining source port entropy, which limits concurrent outbound queries.
• Confirm net.core.rmem_max and related kernel buffer settings are adequate for the query volume.

Forwarder-specific problems

If the resolver uses forwarders (forwarders statement in named.conf), a single slow forwarder can dominate the RTT distribution because all recursive traffic flows through it. Check each forwarder’s health independently. Consider removing or replacing a chronically slow forwarder. BIND will use forwarders in order of RTT if multiple are configured, but a single forwarder configuration has no failover.

EDNS0 and path MTU

If Truncated or EDNS0Fail counters are elevated, the upstream or an intermediate firewall may be dropping EDNS0-advertised packets. Test with reduced buffer sizes: dig @<upstream-ip> example.com +bufsize=512. If responses improve, the path has an MTU or EDNS0 handling issue. Check firewall rules for UDP fragmentation dropping.

Prevention

• Track RTT distribution as a baseline trend. Establish the normal distribution for your resolver and alert on sustained shifts before they cascade.
• Monitor RecursClients as a percentage of the limit. Daily peaks should not exceed 50% of the limit, providing headroom for upstream slowdowns.
• Alert on QueryTimeout rates above 2%. Timed-out upstream queries are the direct precursor to cascade failure.
• Keep recursive-clients adequately sized. The default of 1000 is too low for busy resolvers. Scale it with available file descriptors and memory, but do not raise it without ensuring the system can support the concurrent load.
• Note deprecated and removed options. resolver-nonbackoff-tries and resolver-retry-interval were deprecated in BIND 9.18.21 (using them logs a warning) and removed in BIND 9.20.0, where using them is a fatal error (GL #4405). Do not rely on them for tuning.

How Netdata helps

Netdata’s BIND collector surfaces the per-view QryRTT histogram buckets as per-second metrics, handling the cumulative-to-delta conversion automatically. This eliminates the manual two-sample subtraction that raw statistics channel polling requires.

• Per-second RTT bucket distribution per view shows the current distribution shift in real time, not the all-time cumulative average.
• Correlation with RecursClients (collected as an absolute gauge) shows immediately whether an RTT shift is translating into recursive-client slot pressure.
• Correlation with per-view resolver failure counters (QueryTimeout, Retry, Lame, EDNS0Fail) distinguishes a slow-but-alive upstream from one that is dropping packets.
• Correlation with cache hit ratio reveals whether declining cache effectiveness is amplifying upstream load.
• ML anomaly detection on the RTT distribution can flag a gradual shift before it crosses a fixed threshold.

Related guides

• BIND forwarding loops: recursion that never terminates and burns recursive slots
• How BIND actually works in production: a mental model for operators
• BIND monitoring checklist: the signals every production resolver and authoritative server needs
• BIND monitoring maturity model: from survival to expert
• BIND ’no more recursive clients: quota reached’: the recursive-clients circuit breaker
• named not responding on port 53: total outage versus UDP-works-TCP-fails
• BIND resolver NumFetch per view: per-view recursive pressure in split-horizon setups
• BIND NXDOMAIN spike: DGA malware, water torture, and Windows suffix search lists
• BIND recursive resolution cascade: one slow upstream taking down all resolution
• BIND RecursClients climbing toward the limit: reading the recursive saturation gauge
• BIND REFUSED responses: ACL denials, recursion policy, and clients that get locked out
• rndc not responding: control-plane failure while queries still work