BIND RecursClients climbing toward the limit: reading the recursive saturation gauge

RecursClients is the number of queries currently in flight, each waiting for an upstream nameserver to respond. When this gauge climbs toward the configured recursive-clients limit, BIND is running out of slots to start new recursive lookups. At the hard limit, BIND does not accept new incoming requests and drops an older pending request for each; such new requests receive SERVFAIL.

The metric is a gauge, not a rate. The absolute number means little without the configured ceiling next to it: 300 is comfortable against a limit of 1000 and dangerous against a limit of 350. Track RecursClients as a percentage of recursive-clients, not as a raw count, and watch the daily peak for runway estimation.

The failure curve is cliff-edge. BIND has no graceful degradation between stressed and broken. At the soft quota (90% by default), it accepts the new request but aborts the oldest pending query to make room. At the hard limit, it does not accept the new incoming request and also aborts the oldest pending query; the new request receives SERVFAIL.

What RecursClients actually measures

Every time BIND receives a query that requires recursion (a cache miss for a domain it is configured to resolve), it allocates a recursive client slot. That slot stays occupied until the upstream nameserver responds, the query times out, or the slot is forcibly reclaimed. RecursClients is the instantaneous count of occupied slots.

The value goes up when new recursive queries arrive faster than old ones complete, and down when upstream responses arrive and slots are freed. On an idle server or an authoritative-only server with recursion disabled, RecursClients sits at or near zero. On a busy recursive resolver, it fluctuates with query volume and upstream latency.

Collect it from the statistics channel:

# Check current recursive client count
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  print('RecursClients:', d.get('nsstats',{}).get('RecursClients', 'N/A'))"

The port (8653 in this example) depends on your statistics-channels configuration. The counter lives under the nsstats key in the JSON output and is an absolute gauge value, not a cumulative counter. Zero-valued counters are omitted by default in non-verbose statistics channel output, so a missing key on an idle server usually means zero.

For a detailed view of what is in flight, rndc recursing writes the active recursive queries to named.recursing, including the upstream nameservers they are waiting on. It also reports statistics on active, allowed, and dropped fetch counts due to fetches-per-server and fetches-per-zone quotas. This is the most valuable diagnostic command during an incident because it tells you not just how many slots are occupied but which upstream is causing the pile-up.

The recursive-clients limit and quota mechanics

The recursive-clients option in named.conf caps the number of concurrent recursive queries BIND will process. The default is 1000. BIND enforces this limit in two tiers:

Soft quota. When recursive-clients is 1000 or less (the default), the soft quota is 90% of the configured value: 900 at the default. When recursive-clients is set above 1000, current BIND computes the soft quota as recursive-clients - max(100, worker-thread count + 1). When RecursClients exceeds the soft quota, BIND logs recursive-clients soft limit exceeded (...), aborting oldest query and drops the oldest pending recursive client to make room for each new incoming request. This is backpressure, not a crash.

Hard quota. When RecursClients reaches the configured recursive-clients value, BIND does not accept the new inbound request and also drops the oldest pending request. The new query receives SERVFAIL.

flowchart TD
    A[Upstream nameserver
slows or goes silent] --> B[Each in-flight query
holds a slot for
timeout duration]
    B --> C[Slots accumulate
faster than they drain]
    C --> D[50% of limit:
ticket-level alert]
    D --> E[90% soft quota:
oldest queries aborted
to make room]
    E --> F[100% hard limit:
new requests not accepted,
old pending dropped]

The transition from 90% to 100% is not gradual. Once the hard limit is hit, BIND does not accept new recursive requests, so resolution fails for new queries, including queries targeting healthy upstream nameservers.

Reading the saturation gauge

Operational thresholds expressed as a percentage of the configured recursive-clients limit:

Two rules govern how to use this gauge operationally:

Daily peak should stay under 50% of limit. This provides headroom for traffic spikes, upstream slowdowns, and attack absorption. If your daily peak regularly exceeds 50%, you are operating without a safety margin.

Track the daily peak as a trend for runway estimation. If the daily peak is growing by X% per week, time-to-saturation is approximately (100% - current_peak%) / X% weeks.

Two gating conditions prevent false readings:

• Idle or authoritative-only servers sit near zero because no recursive queries are in flight. This is correct, not a monitoring gap.
• Cold start produces a temporary spike because cache hit rate is 0% after restart and every query triggers recursion. An uptime gate (typically 300 seconds) suppresses alerts during this cache-warming window.

What a rising count tells you

RecursClients climbing means fetches are piling up waiting on upstream. Each in-flight query occupies a slot until the upstream responds or the resolver-query-timeout fires. Under normal conditions, upstream nameservers respond in tens to hundreds of milliseconds, and slots cycle quickly. When upstream degrades, each slot is held longer.

The resolver-query-timeout option controls the total time BIND spends on a single recursive resolution before abandoning it. In BIND 9.18/9.20 it is measured in milliseconds and defaults to 10000; the maximum is 30000. Values at or below 300 are interpreted as seconds and converted to milliseconds. When multiple queries target a slow upstream simultaneously, slots fill rapidly.

During an active climb, rndc recursing is the diagnostic command:

# Have BIND write the current recursing queries to named.recursing,
# then inspect that file and adapt the filter to your BIND version.
rndc recursing
sed -n '1,100p' named.recursing

If most in-flight queries target the same upstream nameserver, the problem is localized. If they are spread across many upstreams, the problem is broader: network path failure, DNSSEC validation overhead, or volume overload.

CPU may not be elevated during a recursive pile-up. Worker threads are blocked waiting for network responses, not computing. A resolver with RecursClients at 85% and low CPU is under upstream latency pressure, not compute pressure. Adding CPU does not help; failing faster or routing around the slow upstream does.

Tuning the limit

The default recursive-clients value of 1000 is adequate for small or medium deployments but too low for busy resolvers. Raising it is a valid response to sustained high utilization, but each slot consumes downstream resources:

• File descriptors. Each recursive query uses one or more file descriptors for outbound sockets. If recursive-clients is set high relative to the FD limit, FD exhaustion occurs before the recursive client limit is reached.
• Memory. BIND documents each recursing client as using on the order of 20 KB. At the default 1000, that is roughly 20 MB; at 10000, roughly 200 MB, before other allocations. Measure RSS after changing the limit.
• clients-per-query and max-clients-per-query. These options (defaults 10 and 100 respectively) control how many identical concurrent recursive queries BIND processes versus deduplicating. Higher clients-per-query means more clients share a single outstanding fetch, reducing slot pressure for popular domains.

Check the currently configured value:

# Show the configured recursive-clients value
named-checkconf -p /etc/named.conf | grep recursive-clients

If no value is printed, BIND is using the default of 1000.

Signals to correlate with RecursClients

RecursClients is the hub of a correlated signal set. Interpret it alongside:

A rising count with a falling cache hit ratio and rising QueryTimeout is the recursive resolution cascade: upstream slowness is filling slots. A rising count with stable cache hit ratio and no upstream signal suggests a volume increase that may warrant capacity expansion.

Version-specific counter reliability

The RecursClients counter has had bugs that corrupt its value:

• BIND 9.18 before 9.18.5 / 9.16 before 9.16.27 (GL #3147): The counter could underflow in certain resolution scenarios.
• BIND 9.18 before 9.18.7 / 9.16 before 9.16.34 (GL #3584): The counter could overflow in certain resolution scenarios.

If your RecursClients value appears negative or implausibly large on one of the affected releases, the counter itself may be corrupt. Any percentage-of-limit alert built on a corrupt counter can fire spuriously or miss real saturation. Upgrade to at least 9.18.7 or 9.16.34 before trusting the metric.

The JSON key RecursClients under nsstats is stable across BIND 9.16 and later.

How Netdata helps

Netdata collects RecursClients as an absolute gauge, so per-second visualization reflects real-time saturation state without rate conversion.

• Percentage-of-limit alerting. Netdata alerts on RecursClients as a percentage of the configured recursive-clients value, applying the 50% warning and 90% critical thresholds automatically. This normalizes across instances with different configured limits.
• Correlation with upstream signals. Netdata dashboards place RecursClients alongside QueryTimeout rates, upstream RTT distributions, and SERVFAIL rates, so cause and effect are visible in one view.
• Cache hit ratio correlation. A declining cache hit ratio is often the earliest predictor of RecursClients pressure. Both metrics share a timeline.
• Uptime gating. Alert rules can incorporate process uptime to suppress false positives during cold-start cache warming.
• Daily peak tracking. Long-term storage retains the daily peak pattern for runway estimation.

Related guides

• How BIND actually works in production: a mental model for operators
• BIND monitoring checklist: the signals every production resolver and authoritative server needs
• BIND monitoring maturity model: from survival to expert
• BIND NXDOMAIN spike: DGA malware, water torture, and Windows suffix search lists
• BIND REFUSED responses: ACL denials, recursion policy, and clients that get locked out
• BIND SERVFAIL responses: what a DNS SERVFAIL actually means and how to trace the cause