BIND 'no more recursive clients: quota reached': the recursive-clients circuit breaker

The log message no more recursive clients (current/soft/hard): quota reached means BIND has run out of slots for in-flight recursive queries, and new queries are failing. Resolution for your clients is already degraded or broken.

The recursive-clients option (default 1000) caps the number of concurrent upstream fetches the resolver can have outstanding. A soft quota at 90 percent (default 900) starts shedding load before the hard limit. Once the hard limit is reached, BIND does not accept new incoming requests and drops a previous pending request for each one; such new requests resolve to SERVFAIL, including queries for healthy upstream nameservers.

The mechanism is a global queue. One slow upstream nameserver can fill the entire queue with queries waiting for timeouts, leaving no room for anything else. This is how a single upstream failure cascades into total local resolution failure.

What this means

The quota mechanism

BIND tracks outstanding recursive queries in the recursive-clients table. The RecursClients statistic is a gauge (current in-flight count), not a cumulative counter.

The limits:

• Soft quota: 90 percent of recursive-clients when the configured value is 1000 or less. For default configuration, that is 900. When recursive-clients is set above 1000, current BIND computes the soft quota as the hard limit minus max(100, worker-thread count + 1).
• Hard limit: The value of recursive-clients, default 1000.

At the soft quota, BIND begins shedding. For each new incoming recursive query, it drops the oldest pending query to make room. The new query is accepted, but the dropped query is a real loss for the client that sent it.

At the hard limit, BIND does not accept the new incoming request and also drops the oldest pending query. In the normal query path, the new request is answered with SERVFAIL.

rndc status shows the three numbers

rndc status displays recursive client utilization as three values:

recursive clients: <current>/<soft_limit>/<hard_limit>

With default configuration and an idle server, this reads 0/900/1000. The second number is the soft limit, not a second count. Operators frequently misread this as “900 active clients.”

The cascade

Each in-flight recursive query holds its slot until the upstream responds or the resolver query timeout expires. resolver-query-timeout is the total time BIND spends attempting one recursive resolution; its BIND 9.20 default is 10000 ms and bounds the complete resolution, including retries and attempts to multiple authoritative servers.

flowchart TD
    A[Slow or unreachable upstream NS] --> B[Each query to that NS holds a slot]
    B --> C[Slots accumulate: 100, 500, 900]
    C --> D[Soft quota 900 reached]
    D --> E[BIND drops oldest pending query per new one]
    E --> F[Queries to healthy upstreams also get dropped]
    F --> G[Hard limit 1000 reached]
    G --> H[New requests not accepted; old pending dropped]

Common causes

Quick checks

# Check current recursive client utilization (current/soft/hard)
rndc status | grep "recursive clients"

# Ask BIND to write its current recursing queries to named.recursing,
# then inspect that file (the path is normally the working directory).
rndc recursing
sed -n '1,30p' named.recursing

# Count the query records in the dump; the format varies by BIND version,
# so inspect the file before adapting this filter.
grep '^; ; Active fetch domains' -n named.recursing

# Check RecursClients from the JSON v1 statistics channel.
# Adjust the port to match your named.conf statistics-channels block.
curl -s http://localhost:8053/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  print('RecursClients:', d.get('nsstats',{}).get('RecursClients','N/A'))"

# Check resolver timeout counters per view
curl -s http://localhost:8053/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(f'{v}: {k}={s}') for v,vd in d.get('views',{}).items() \
  for k,s in vd.get('resolver',{}).get('stats',{}).items() if 'Timeout' in k]"

# Verify resolver-query-timeout and recursive-clients settings
# named-checkconf without a path reads the default config file location
named-checkconf -p | grep -iE "resolver-query-timeout|recursive-clients"

# Check SERVFAIL rate
curl -s http://localhost:8053/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  print('QrySERVFAIL:', d.get('nsstats',{}).get('QrySERVFAIL','N/A'))"

# Verify BIND version for known bugs
named -v

How to diagnose it

1. Read the current utilization. Run rndc status | grep "recursive clients". If the first number is near or above the second (soft quota), you are in the danger zone. If it is at the third number (hard limit), you are actively failing.

2. Identify the offending upstream. Run rndc recursing, then inspect named.recursing for the upstream nameserver IP that appears most frequently. A concentration against one or a small set of IPs identifies the bottleneck.

3. Confirm the failure pattern. Check whether SERVFAIL is broad (affecting many unrelated domains) or narrow (specific zones). Broad SERVFAIL with RecursClients near the limit confirms the cascade pattern. If SERVFAIL is zone-specific, the problem is elsewhere: zone load failure, DNSSEC validation failure, or broken delegation.

4. Check resolver timeout counters. Rising QueryTimeout in per-view resolver stats confirms upstream unreachability. Compare the timeout rate against the RTT distribution buckets. If QryRTT1600+ is elevated, upstream is slow rather than dead.

5. Verify it is not a cold start. If uptime is under 30 minutes and cache hit ratio is near 0%, the quota pressure is from cache warming. This is expected and self-corrects as the cache fills.

6. Check for known version bugs. BIND 9.16 before 9.16.27 and BIND 9.18 before 9.18.5 could underflow the RecursClients counter in some resolution scenarios; BIND 9.16 before 9.16.34 and BIND 9.18 before 9.18.7 could overflow it. If named crashes alongside quota messages, check your version. Additionally, stale-answer-client-timeout greater than 0 could crash named when the soft quota was reached; BIND 9.18.11 fixed that CVE-2022-3924 crash. In BIND 9.20.0, non-zero stale-answer-client-timeout values generate a warning and are treated as 0, removing the crash path but also disabling that timeout.

Metrics and signals to monitor

Fixes

Immediate mitigation during an incident

Flush the affected domain’s cache entries. If named.recursing shows queries piling up against a specific upstream, flushing that domain’s cache can clear stuck entries:

# WARNING: flushing forces clients to re-query, temporarily increasing load.
# Flush only the specific name causing the bottleneck, not the entire cache.
rndc flushname example.com

This is temporary. If the upstream is still slow, queries will pile up again.

Lower resolver-query-timeout to fail fast. The default is 10000 ms. Lowering it reduces the time each failing query holds a slot, freeing slots sooner for other queries. BIND 9.20 accepts 301-30000 ms; values at or below 300 are interpreted as seconds and converted. Changing this requires a configuration edit and rndc reconfig. The tradeoff: faster failure for stuck queries versus false negatives for legitimate slow resolution.

Raise the recursive-clients limit

If the daily peak consistently approaches the soft quota during normal traffic (no upstream failure), the configured limit is too low for the workload:

options {
    recursive-clients 5000;
};

BIND documents each recursive client as on the order of 20 KB of memory. Raising from 1000 to 5000 adds roughly 80 MB before other allocations; measure actual RSS after the change. Ensure the file descriptor limit is sufficient, since each in-flight recursive query uses one or more file descriptors:

# Current FD usage vs limit
ls /proc/$(pgrep -x named)/fd | wc -l
grep "Max open files" /proc/$(pgrep -x named)/limits

The files option is deprecated in BIND 9.18 and removed in 9.20. File descriptor limits are controlled by the OS (ulimit, systemd LimitNOFILE).

Address the root cause upstream

If a specific upstream nameserver is consistently slow or unreachable:

• Verify network path: dig @<upstream-ip> example.com SOA +time=2 +tries=1
• Check if the upstream is rate-limiting your resolver
• Consider removing the problematic forwarder from the forwarders list if forwarder mode is configured
• For authoritative upstreams you do not control, BIND’s RTT-based server selection will eventually deprioritize slow servers, but only after observing the slowness through actual queries

Tune query deduplication

clients-per-query (default 10) and max-clients-per-query (default 100) control how many clients can wait for the same recursive query simultaneously. BIND deduplicates identical concurrent queries so only one upstream fetch is performed. Lowering clients-per-query causes excess clients for a popular query to receive immediate failures rather than waiting, which frees slots for other queries. Raising it reduces upstream load for popular domains at the cost of more clients waiting per slot. The right values depend on your query mix.

Prevention

Monitor RecursClients as a percentage of the configured limit. This is the single most important preventive signal. Normal operation should keep the daily peak below 50 percent of the limit, providing headroom for traffic spikes, upstream slowdowns, and attack absorption. If the daily peak is growing week over week, estimate time-to-saturation: (limit - current_peak) / weekly_growth.

Track upstream RTT trends. Rising RTT toward upstream nameservers is a leading indicator. If the RTT distribution shifts toward higher buckets before the daily peak, upstream degradation is underway. The QryRTT1600+ bucket is the severely degraded signal.

Monitor cache hit ratio. A falling hit ratio means more outbound queries, which means more concurrent recursive clients. Any condition that degrades cache effectiveness (restart, flush, traffic pattern change, undersized cache) directly increases recursive client pressure.

Verify your BIND version is patched. If running BIND 9.16.x or 9.18.x, confirm you are on or above 9.16.27 or 9.18.5 to avoid the RecursClients underflow. If running with stale-answer-client-timeout > 0, confirm you are on or above 9.18.11.

Use BIND 9.20+ if possible. The recursive high-water statistic, added in BIND 9.20.0, reports the maximum number of simultaneous recursive clients seen since named started. This is valuable for capacity planning: if the high-water mark is 850 against a limit of 1000, you came close even if the current count has since dropped.

How Netdata helps

Netdata collects BIND statistics channel data at per-second resolution, which matters for this failure pattern because the cascade from stressed to broken can happen in seconds.

• RecursClients as a gauge: Netdata collects RecursClients as an absolute value. Pairing this with the configured recursive-clients limit as a dashboard threshold makes the approach to the quota visible before it trips.
• Correlation with SERVFAIL: When RecursClients climbs toward the limit and QrySERVFAIL begins rising simultaneously, the composite pattern confirms the cascade without manual cross-referencing.
• Per-view resolver stats: QueryTimeout and NumFetch per view reveal which view is experiencing upstream pressure. This matters in split-horizon deployments where aggregate statistics mask view-specific problems.
• RTT distribution tracking: Shifts in QryRTT buckets provide early warning that upstream nameservers are slowing down, before RecursClients begins to climb.

Related guides

• How BIND actually works in production: a mental model for operators
• BIND monitoring checklist: the signals every production resolver and authoritative server needs
• BIND monitoring maturity model: from survival to expert
• BIND NXDOMAIN spike: DGA malware, water torture, and Windows suffix search lists
• BIND REFUSED responses: ACL denials, recursion policy, and clients that get locked out
• BIND SERVFAIL responses: what a DNS SERVFAIL actually means and how to trace the cause