BIND 'no more recursive clients: quota reached': the recursive-clients circuit breaker

The log message no more recursive clients (N/M): quota reached means BIND has run out of slots for in-flight recursive queries, and new queries are failing. Resolution for your clients is already degraded or broken.

The recursive-clients option (default 1000) caps the number of concurrent upstream fetches the resolver can have outstanding. A soft quota at 90 percent (default 900) starts shedding load before the hard limit. Once the hard limit is reached, every new recursive query returns SERVFAIL, including queries for domains whose upstream nameservers are healthy.

The mechanism is a global queue. One slow upstream nameserver can fill the entire queue with queries waiting for timeouts, leaving no room for anything else. This is how a single upstream failure cascades into total local resolution failure.

What this means

The quota mechanism

BIND tracks outstanding recursive queries in the recursive-clients table. The RecursClients statistic is a gauge (current in-flight count), not a cumulative counter.

The limits:

• Soft quota: 90 percent of recursive-clients when the configured value is 1000 or less. For default configuration, that is 900. When recursive-clients is set above 1000, the soft quota is the hard limit minus max(100, number of worker threads).
• Hard limit: The value of recursive-clients, default 1000.

At the soft quota, BIND begins shedding. For each new incoming recursive query, it drops the oldest pending query to make room. The new query is accepted, but the dropped query is a real loss for the client that sent it.

At the hard limit, BIND refuses the new query with SERVFAIL.

rndc status shows the three numbers

rndc status displays recursive client utilization as three values:

recursive clients: <current>/<soft_limit>/<hard_limit>

With default configuration and an idle server, this reads 0/900/1000. The second number is the soft limit, not a second count. Operators frequently misread this as “900 active clients.”

The cascade

Each in-flight recursive query holds its slot until the upstream responds or the resolver query timeout expires. The default resolver-query-timeout is 10 seconds total for the entire resolution, including retries and multiple authoritative server attempts.

flowchart TD
    A[Slow or unreachable upstream NS] --> B[Each query to that NS holds a slot]
    B --> C[Slots accumulate: 100, 500, 900]
    C --> D[Soft quota 900 reached]
    D --> E[BIND drops oldest pending query per new one]
    E --> F[Queries to healthy upstreams also get dropped]
    F --> G[Hard limit 1000 reached]
    G --> H[ALL new recursive queries return SERVFAIL]

Common causes

Quick checks

# Check current recursive client utilization (current/soft/hard)
rndc status | grep "recursive clients"

# See what queries are in flight and which upstream they wait on
rndc recursing | head -30

# Count in-flight queries by upstream nameserver
# Note: output format varies by BIND version; adjust parsing accordingly
rndc recursing -j 2>/dev/null | head -100 || rndc recursing | awk '{print $NF}' | sort | uniq -c | sort -rn | head -10

# Check RecursClients from the statistics channel
# Note: adjust the port and URL path to match your named.conf statistics-channels block
curl -s http://localhost:8053/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  print('RecursClients:', d.get('nsstats',{}).get('RecursClients','N/A'))"

<!-- TODO: verify the JSON v1 statistics channel endpoint path and whether RecursClients is under nsstats for BIND 9.18+. -->

# Check resolver timeout counters per view
curl -s http://localhost:8053/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(f'{v}: {k}={s}') for v,vd in d.get('views',{}).items() \
  for k,s in vd.get('resolver',{}).get('stats',{}).items() if 'Timeout' in k]"

# Verify resolver-query-timeout and recursive-clients settings
# named-checkconf without a path reads the default config file location
named-checkconf -p | grep -iE "resolver-query-timeout|recursive-clients"

# Check SERVFAIL rate
curl -s http://localhost:8053/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  print('QrySERVFAIL:', d.get('nsstats',{}).get('QrySERVFAIL','N/A'))"

# Verify BIND version for known bugs
named -v

How to diagnose it

1. Read the current utilization. Run rndc status | grep "recursive clients". If the first number is near or above the second (soft quota), you are in the danger zone. If it is at the third number (hard limit), you are actively failing.

2. Identify the offending upstream. Run rndc recursing and look for which upstream nameserver IP appears most frequently. A concentration against one or a small set of IPs identifies the bottleneck.

3. Confirm the failure pattern. Check whether SERVFAIL is broad (affecting many unrelated domains) or narrow (specific zones). Broad SERVFAIL with RecursClients near the limit confirms the cascade pattern. If SERVFAIL is zone-specific, the problem is elsewhere: zone load failure, DNSSEC validation failure, or broken delegation.

4. Check resolver timeout counters. Rising QueryTimeout in per-view resolver stats confirms upstream unreachability. Compare the timeout rate against the RTT distribution buckets. If QryRTT1600+ is elevated, upstream is slow rather than dead.

5. Verify it is not a cold start. If uptime is under 30 minutes and cache hit ratio is near 0%, the quota pressure is from cache warming. This is expected and self-corrects as the cache fills.

6. Check for known version bugs. BIND 9.16.x and 9.18.x before 9.18.7 had a bug where the RecursClients gauge could underflow (go negative), causing an assertion failure crash. If named is crashing in addition to quota messages, check your version. The fix landed in 9.16.27, 9.18.7, and was further refined in 9.18.31. Additionally, stale-answer-client-timeout set to a value greater than 0 could crash named when the soft quota was reached. This was fixed in 9.18.12 (and 9.16.38). In BIND 9.20.0, non-zero stale-answer-client-timeout values are silently treated as 0, removing the crash path but also disabling the feature.

Metrics and signals to monitor

Fixes

Immediate mitigation during an incident

Flush the affected domain’s cache entries. If rndc recursing shows queries piling up against a specific upstream, flushing that domain’s cache can clear stuck entries:

# WARNING: rndc flush forces clients to re-query, temporarily increasing load.
# Flush only the specific domain causing the bottleneck, not the entire cache.
rndc flush example.com

This is temporary. If the upstream is still slow, queries will pile up again.

Lower resolver-query-timeout to fail fast. The default is 10 seconds. Lowering it reduces the time each failing query holds a slot, freeing slots sooner for other queries. Changing this requires a configuration edit and rndc reconfig. The tradeoff: faster failure for stuck queries versus false negatives for legitimate slow resolution.

Raise the recursive-clients limit

If the daily peak consistently approaches the soft quota during normal traffic (no upstream failure), the configured limit is too low for the workload:

options {
    recursive-clients 5000;
};

Each recursive client slot consumes approximately 20 KB of memory. Raising from 1000 to 5000 adds roughly 80 MB. Ensure the file descriptor limit is sufficient, since each in-flight recursive query uses one or more file descriptors:

# Current FD usage vs limit
ls /proc/$(pgrep -x named)/fd | wc -l
grep "Max open files" /proc/$(pgrep -x named)/limits

The files option is deprecated in BIND 9.18 and removed in 9.20. File descriptor limits are controlled by the OS (ulimit, systemd LimitNOFILE).

Address the root cause upstream

If a specific upstream nameserver is consistently slow or unreachable:

• Verify network path: dig @<upstream-ip> example.com SOA +time=2 +tries=1
• Check if the upstream is rate-limiting your resolver
• Consider removing the problematic forwarder from the forwarders list if forwarder mode is configured
• For authoritative upstreams you do not control, BIND’s RTT-based server selection will eventually deprioritize slow servers, but only after observing the slowness through actual queries

Tune query deduplication

clients-per-query (default 10) and max-clients-per-query (default 100) control how many clients can wait for the same recursive query simultaneously. BIND deduplicates identical concurrent queries so only one upstream fetch is performed. Lowering clients-per-query causes excess clients for a popular query to receive immediate failures rather than waiting, which frees slots for other queries. Raising it reduces upstream load for popular domains at the cost of more clients waiting per slot. The right values depend on your query mix.

Prevention

Monitor RecursClients as a percentage of the configured limit. This is the single most important preventive signal. Normal operation should keep the daily peak below 50 percent of the limit, providing headroom for traffic spikes, upstream slowdowns, and attack absorption. If the daily peak is growing week over week, estimate time-to-saturation: (limit - current_peak) / weekly_growth.

Track upstream RTT trends. Rising RTT toward upstream nameservers is a leading indicator. If the RTT distribution shifts toward higher buckets before the daily peak, upstream degradation is underway. The QryRTT1600+ bucket is the severely degraded signal.

Monitor cache hit ratio. A falling hit ratio means more outbound queries, which means more concurrent recursive clients. Any condition that degrades cache effectiveness (restart, flush, traffic pattern change, undersized cache) directly increases recursive client pressure.

Verify your BIND version is patched. If running BIND 9.16.x or 9.18.x, confirm you are on or above 9.16.27 or 9.18.7 to avoid the RecursClients underflow crash. If running with stale-answer-client-timeout > 0, confirm you are on or above 9.18.12.

Use BIND 9.20+ if possible. The recursive high-water statistic, added in BIND 9.20.0, reports the maximum number of simultaneous recursive clients seen since named started. This is valuable for capacity planning: if the high-water mark is 850 against a limit of 1000, you came close even if the current count has since dropped.

How Netdata helps

Netdata collects BIND statistics channel data at per-second resolution, which matters for this failure pattern because the cascade from stressed to broken can happen in seconds.

• RecursClients as a gauge: Netdata collects RecursClients as an absolute value. Pairing this with the configured recursive-clients limit as a dashboard threshold makes the approach to the quota visible before it trips.
• Correlation with SERVFAIL: When RecursClients climbs toward the limit and QrySERVFAIL begins rising simultaneously, the composite pattern confirms the cascade without manual cross-referencing.
• Per-view resolver stats: QueryTimeout and NumFetch per view reveal which view is experiencing upstream pressure. This matters in split-horizon deployments where aggregate statistics mask view-specific problems.
• RTT distribution tracking: Shifts in QryRTT buckets provide early warning that upstream nameservers are slowing down, before RecursClients begins to climb.

Related guides

• How BIND actually works in production: a mental model for operators
• BIND monitoring checklist: the signals every production resolver and authoritative server needs
• BIND monitoring maturity model: from survival to expert
• BIND NXDOMAIN spike: DGA malware, water torture, and Windows suffix search lists
• BIND REFUSED responses: ACL denials, recursion policy, and clients that get locked out
• BIND SERVFAIL responses: what a DNS SERVFAIL actually means and how to trace the cause