BIND (named) processes queries through a pipeline: receive packet, parse, ACL/view check, cache lookup (if recursive), zone lookup (if authoritative), recursive fetch on cache miss, apply RPZ/DNSSEC, serialize response, send. Every signal below maps to a stage in that pipeline or a resource it competes for: CPU, memory, file descriptors, network buffers, source ports.
The levels are cumulative. Most signals come from the statistics channel (JSON at /json/v1/server, explicitly configured in named.conf). The rndc stats file is an alternative but appends indefinitely and can fill disk if not rotated. Commands below assume a single named process (standard deployment; BIND is multithreaded, not multiprocess).
The maturity framework
flowchart TD
L1["Level 1: Survival - is it alive?"]
L2["Level 2: Operational - degradation detection"]
L3["Level 3: Mature - leading indicators"]
L4["Level 4: Expert - per-thread, per-upstream"]
L1 --> L2 --> L3 --> L4- Level 1: Survival - Is
namedalive and answering? Catches total outages only. - Level 2: Operational - Common degradation before it becomes an outage. Minimum for production.
- Level 3: Mature - Leading indicators and internal state. Problems visible hours to days before user impact.
- Level 4: Expert - Per-thread, per-view, per-upstream granularity. Added after major incidents.
Level 1: Survival
| Signal | Why it matters | How to check |
|---|---|---|
| Process liveness | If named is gone, DNS fails for all clients | pgrep -x named (use -x, not -f, to avoid false matches on named-checkzone) |
| Functional UDP query (role-correct) | UDP carries 95%+ of DNS traffic | Recursive: dig +time=2 +tries=1 @127.0.0.1 example.com A. Authoritative: dig +time=2 +tries=1 +norecurse @127.0.0.1 <your-zone> SOA |
| Functional TCP query | TCP carries zone transfers, large responses, DNSSEC | Same probe with +tcp |
| Process memory (RSS) | Unbounded growth leads to OOM kill | awk '/VmRSS/{print $2}' /proc/$(pgrep -x named)/status (value in kB) |
A REFUSED response means ACL denial, not service failure. An authoritative server returns REFUSED for queries outside its zones. The canaries above already account for this.
Level 2: Operational
| Signal | Why it matters | How to check |
|---|---|---|
| Incoming query rate | Baseline traffic. Sudden drop = listener failure or network partition. Spike = DDoS or flash crowd | Requestv4, Requestv6 in NSStats via statistics channel |
| SERVFAIL rate | Server is up but failing to resolve. Most important error signal | QrySERVFAIL / (QrySuccess + QrySERVFAIL + QryNXDOMAIN + QryFORMERR + QryNxrrset + QryReferral). Normal: near 0%. Alert: >0.1%. Critical: >1% |
| Recursive clients in-flight | BIND’s circuit breaker. Soft quota at 90% of limit (default 900). Hard limit at 100% (default 1000) = all new recursive queries get SERVFAIL | RecursClients in NSStats as % of recursive-clients config. Normal: <30%. Alert: >50%. Critical: >90% |
| Cache hit ratio | Low hit ratio = high latency, more upstream load, beginning of recursive pain | CacheHits / (CacheHits + CacheMisses) per view. Should be >90% after warmup (30-60 min post-restart) |
| Protocol distribution | TCP share normally <5%. Elevation indicates truncation, transfers, or attacks | QryUDP vs QryTCP in NSStats |
| Query rejection rate | ACL denials from legitimate clients indicate misconfiguration | AuthQryRej, RecQryRej in NSStats |
| CPU utilization | DNSSEC validation, query parsing, RPZ matching all consume CPU | pidstat -p $(pgrep -x named) or /proc/$(pgrep -x named)/stat |
| File descriptor usage | FD exhaustion causes silent query drops. Default ulimit -n (often 1024) is dangerously low | ls /proc/$(pgrep -x named)/fd | wc -l vs grep "Max open files" /proc/$(pgrep -x named)/limits. Peak should not exceed 50% of limit |
| Zone transfer / SOA serial | Secondaries serving stale data is a ticking bomb. Serial mismatch persisting beyond refresh interval indicates transfer failure | dig @primary <zone> SOA +short vs dig @secondary <zone> SOA +short (compare serial, third field) |
| DNSSEC validation failures | ValFail produces SERVFAIL for signed domains. Broad failures across unrelated domains usually mean local problem | ValFail in per-view resolver stats. Should be 0 or near-0 |
| Zone load health after reload | named starts successfully even if zones fail to load. rndc status reports “running” but the zone does not work | rndc status | grep -i zones for zone count, then journalctl -u named --since "5 min ago" | grep -i "zone.*loaded|zone.*failed" (service name is bind9 on Debian/Ubuntu) |
SERVFAIL is cached (negative caching). A momentary upstream outage causes sustained SERVFAIL for the negative TTL duration. Fixing upstream does not instantly fix the metric. On public recursive resolvers, some broken external domains produce background SERVFAIL. Breadth and trend matter more than individual occurrences.
Level 3: Mature
| Signal | Why it matters | How to check |
|---|---|---|
| Resolver RTT distribution | Upstream latency directly drives cache-miss latency for users. Shift toward higher buckets indicates upstream degradation | QryRTT10, QryRTT100, QryRTT500, QryRTT800, QryRTT1600, QryRTT1600+ per view |
| Resolver failure counters | QueryTimeout holds a recursive-client slot for up to resolver-query-timeout seconds (default 10s) per timed-out query | QueryTimeout, Lame, QuerySockFail, QueryAbort, Retry per view. Normal: <2% timeouts. Alert: >5%. Critical: >20% |
| Active resolver queries per view | Per-view version of recursive workload pressure | NumFetch per view (gauge, not counter) |
| Response code breakdown | NXDOMAIN spike may indicate DGA malware or water torture attack | NOERROR, NXDOMAIN, SERVFAIL, REFUSED individually from rcodes section |
| UDP RcvbufErrors | Packets dropped by kernel before BIND sees them. Invisible to BIND statistics. The single most systematic monitoring gap | cat /proc/net/snmp | grep Udp (UdpRcvbufErrors column). Any sustained non-zero rate is abnormal |
| TCP connection count | TCP accumulation can exhaust FDs. Hidden by UDP-only health checks | ss -tan '( sport = :53 )' |
| RRL activity | Rate limiting affecting legitimate traffic. RateSlipped causes TCP retry | RateDropped, RateSlipped in NSStats |
| RPZ rewrites | Spike indicates malware outbreak or botnet activity | RPZRewrites in NSStats |
| Update activity | UpdateFail spikes indicate unauthorized update attempts or broken automation | UpdateDone, UpdateFail in NSStats |
| Socket statistics | Socket activity patterns for FD exhaustion diagnosis | SockStats counters |
| OpCode and QType distribution | ANY query spikes indicate amplification attacks. TXT spikes may indicate DNS tunneling | opcodes and qtypes maps |
| SOA expire countdown | Real danger signal for secondaries. At expiry, secondary stops serving the zone entirely | rndc zonestatus <zone> shows expire time. Alert when below 50%. Critical when below 25% or 24 hours |
| Cache eviction counters | DeleteLRU increasing rapidly indicates cache at capacity and evicting entries | DeleteLRU, DeleteTTL in per-view cachestats |
| Control-plane responsiveness | If rndc hangs while queries work, incident response is impaired | timeout 5 rndc status >/dev/null 2>&1 && echo OK || echo FAIL |
Level 4: Expert
| Signal | Why it matters | How to check |
|---|---|---|
| Per-thread CPU utilization | Single saturated thread bottlenecks whole server while aggregate CPU looks moderate | pidstat -t -p $(pgrep -x named) |
rndc recursing sampling | Shows which upstream nameservers are causing pile-up during recursive resolution cascade | rndc recursing | awk '{print $NF}' | sort | uniq -c | sort -rn | head |
| Cache memory tracking | TreeMemInUse and HeapMemInUse trending up indicates memory pressure before OOM | TreeMemInUse, HeapMemInUse, CacheNodes in cachestats |
| RRSIG expiry monitoring | Silent signing failure leads to worldwide SERVFAIL when signatures expire. Authoritative server does not validate its own signatures | dig @127.0.0.1 <zone> RRSIG +dnssec +multiline for validity window. Alert when below 25% of signature lifetime |
| Query name entropy | High entropy (random subdomains) indicates water torture attack. Near-zero repetition per unique query name | Query log sampling or rndc dumpdb -cache analysis (warning: expensive on large caches, causes I/O pressure) |
| NTP clock offset | Clock drift causes DNSSEC validation failures | timedatectl status or chronyc tracking |
| Per-core CPU and IRQ/softirq | Single-core bottleneck under high packets-per-second load. Bandwidth may have headroom (pps limit, not bps) | mpstat -P ALL 1 5 |
| Source port entropy | Poor randomization = cache poisoning vulnerability | dig +short porttest.dns-oarc.net TXT @127.0.0.1 |
| Statistics channel response time | Slow polling = BIND under severe internal pressure | Time the curl request to statistics channel |
| Zone file integrity checksums | Detect unauthorized zone modifications | Checksum comparison after each reload |
| Post-reload zone load verification | Catches zones that failed to load after rndc reload. named continues running but the zone does not work | rndc zonestatus <zone> for each zone, or log scan for load failures |
| DNSSEC signing freshness | Missing .signed.jnl = zone not signed (silent failure, no log error). Key file missing produces no log error | ls -la /var/named/data/<zone>.signed.jnl 2>/dev/null || echo "NOT SIGNED" (path varies by distribution) |
Role-specific priorities
Recursive resolvers: prioritize cache hit ratio, recursive clients as percentage of limit, upstream RTT distribution, DNSSEC validation failures, and UDP RcvbufErrors. The cache is the dominant memory consumer and the primary value the resolver provides.
Authoritative-only servers: prioritize zone load health, transfer state, SOA serial and expire countdown, DNSSEC signing freshness for inline-signed zones, and TCP behavior. Zone data can dwarf cache memory on servers with many zones.
Mixed-role deployments (recursive and authoritative on the same instance) are the hardest to reason about. Recursive and authoritative failure signals mask each other in aggregate statistics. Per-view monitoring is essential. A zone-specific authoritative failure can hide behind recursive traffic noise.
Common blind spots
Kernel-level UDP drops. BIND statistics only count queries it read from the socket. Queries dropped by the kernel (receive buffer overflow) are invisible to BIND: no log, no counter. The only evidence is UdpRcvbufErrors in /proc/net/snmp. Most teams discover this during an incident where BIND is healthy but queries are disappearing.
Recursive-clients as a cliff edge. This is BIND’s circuit breaker. When it trips, every recursive query gets SERVFAIL. The soft quota at 90% (default 900) starts rejecting before the hard limit (1000). Monitor RecursClients as a percentage of the configured limit, not an absolute number.
SOA expire runway. Zone transfer failures are silent and time-delayed. Secondaries serve stale data until the SOA expire timer runs out, then SERVFAIL. The danger is not serial mismatch alone but the expire countdown to zero.
DNSSEC signature expiry for authored zones. Auto-signing can silently fail (key file permissions, disk full). Missing key files produce no log error and no .signed.jnl file. The authoritative server does not validate its own signatures. Validating resolvers worldwide reject the zone when signatures expire.
How Netdata helps
- Collects
RecursClientsas an absolute gauge, not a cumulative counter, so you see real-time recursive client utilization without computing deltas. - Correlates the recursive resolution cascade in one view:
RecursClientsclimbing toward limit,QueryTimeoutincreasing,QrySERVFAILrising, cache hit ratio declining. These four signals together confirm upstream slowness cascading into local failure. - Surfaces OS-level signals (
UdpRcvbufErrorsfrom/proc/net/snmp, FD usage from/proc/<pid>/fd, per-core CPU) alongside BIND statistics channel data, closing the gap between kernel-level packet drops and BIND-level query counters. - Provides per-view cache hit ratio and eviction counters (
DeleteLRU,DeleteTTL) for split-horizon deployments where aggregate statistics mask view-specific problems. - Excludes the internal
BucketSizestat from RTT histograms automatically. - Handles statistics channel polling at safe intervals with rate computation, avoiding the overhead that polling too frequently (under 5 seconds) adds on busy resolvers.
Related guides
- How BIND actually works in production: a mental model for operators
- BIND monitoring maturity model: from survival to expert
- BIND SERVFAIL responses: what a DNS SERVFAIL actually means and how to trace the cause
- BIND REFUSED responses: ACL denials, recursion policy, and clients that get locked out
- named not responding on port 53: total outage versus UDP-works-TCP-fails
- BIND NXDOMAIN spike: DGA malware, water torture, and Windows suffix search lists
- rndc not responding: control-plane failure while queries still work
- BIND ’no more recursive clients: quota reached’: the recursive-clients circuit breaker
- BIND RecursClients climbing toward the limit: reading the recursive saturation gauge
- BIND recursive resolution cascade: one slow upstream taking down all resolution
- BIND forwarding loops: recursion that never terminates and burns recursive slots
- BIND resolver NumFetch per view: per-view recursive pressure in split-horizon setups