BIND SERVFAIL responses: what a DNS SERVFAIL actually means and how to trace the cause

SERVFAIL (RCODE 2) means BIND attempted resolution and could not return a usable answer. It is not NXDOMAIN (name does not exist), REFUSED (policy denial), or FORMERR (malformed query). The process is alive, port 53 is open, the query was received, and the answer is failure.

This makes SERVFAIL invisible to binary health checks. A BIND server returning 100% SERVFAIL to every client still passes process-liveness and port-check probes. SERVFAIL is a symptom with many possible causes: upstream nameserver timeouts, DNSSEC validation failures, recursive-clients exhaustion, broken delegation, a zone that failed to load, or a configuration error. Tracing it requires correlating multiple BIND signals.

What this means

BIND tracks SERVFAIL responses through the QrySERVFAIL counter in NSStats, exposed via the statistics channel. The counter is cumulative since process start. Express it as a ratio of classified responses to make it useful:

SERVFAIL ratio = QrySERVFAIL / (QrySuccess + QrySERVFAIL + QryNXDOMAIN + QryFORMERR + QryNxrrset + QryReferral)

SERVFAIL is negatively cached. The servfail-ttl option (default 1 second in modern BIND versions ) controls how long SERVFAIL responses are cached. A momentary upstream outage can produce sustained SERVFAIL for the negative TTL duration. Fixing the upstream does not instantly fix the metric. After recovery, you must wait for negative cache entries to expire before the SERVFAIL rate drops.

On public recursive resolvers, some broken external domains produce background SERVFAIL continuously. Breadth and trend matter more than individual occurrences. A SERVFAIL rate of 0.05% concentrated against one broken domain is less urgent than 0.05% spread across many unrelated domains.

Common causes

Quick checks

The following commands assume the statistics channel is configured on 127.0.0.1:8653. Adjust the port to match your statistics-channels block. All commands are read-only unless noted.

# Check SERVFAIL ratio from statistics channel
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); ns=d.get('nsstats',{}); \
  sf=ns.get('QrySERVFAIL',0); \
  denom=sum(ns.get(k,0) for k in ('QrySuccess','QrySERVFAIL','QryNXDOMAIN','QryFORMERR','QryNxrrset','QryReferral')); \
  print(f'QrySERVFAIL: {sf}  ratio: {sf/max(denom,1)*100:.3f}%')"

# Check recursive client utilization (the circuit breaker)
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  print('RecursClients:', d.get('nsstats',{}).get('RecursClients','N/A'))"

# See which upstream nameservers queries are stuck waiting on
rndc recursing | awk '{print $NF}' | sort | uniq -c | sort -rn | head -10

# Test DNSSEC: if +cd works but plain dig fails, DNSSEC validation is the cause
dig +time=2 +tries=1 @127.0.0.1 example.com A +cd +short
dig +time=2 +tries=1 @127.0.0.1 example.com A +short

# Check DNSSEC validation counters per view
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(f'{v}: {k}={s}') for v,vd in d.get('views',{}).items() \
  for k,s in vd.get('resolver',{}).get('stats',{}).items() if k.startswith('Val')]"

# Check resolver timeout counters (upstream not responding)
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(f'{v}: {k}={s}') for v,vd in d.get('views',{}).items() \
  for k,s in vd.get('resolver',{}).get('stats',{}).items() \
  if k in ('QueryTimeout','Lame','QuerySockFail','QueryAbort','Retry')]"

# Check zone load health
rndc status | grep -i "zones"
journalctl -u named --since "5 min ago" | grep -i "zone.*loaded\|zone.*failed\|not loaded"

# Check trust anchor state
rndc managed-keys status

On Debian/Ubuntu, the systemd unit is typically bind9, not named. Substitute journalctl -u bind9 accordingly.

How to diagnose it

The diagnostic flow branches on whether SERVFAIL is broad (many domains) or zone-specific, and whether recursive client pressure is involved.

flowchart TD
    A["SERVFAIL rate elevated"] --> B{"Broad or zone-specific?"}
    B -- "Broad, many domains" --> C{"RecursClients near limit?"}
    B -- "Zone-specific" --> D["Check zone load status"]
    C -- "Yes, above 90%" --> E["Recursive cascade\nfrom upstream timeout"]
    C -- "No" --> F{"ValFail rising?"}
    F -- "Yes" --> G["DNSSEC validation failure"]
    F -- "No" --> H["Check upstream RTT\nand QueryTimeout"]
    D --> I{"Zone loaded?"}
    I -- "No" --> J["Zone load failure"]
    I -- "Yes" --> K["Check delegation\nand transfer health"]

Step 1: Classify the scope.

Determine whether SERVFAIL is affecting many unrelated domains or is limited to specific zones. Broad SERVFAIL points to a systemic issue: upstream cascade, DNSSEC validation, or recursive-clients exhaustion. Zone-specific SERVFAIL points to zone load failure, broken delegation, or zone transfer expiry.

Step 2: Check the recursive client pressure gauge.

RecursClients is a gauge, not a counter. The recursive-clients option (default 1000) caps it. A soft quota at 90% (default 900) means BIND starts rejecting new recursive queries beyond that point. At the hard limit, all new recursive queries get SERVFAIL. If RecursClients is near the limit, you are in a recursive resolution cascade.

Run rndc recursing to see which upstream nameservers the stuck queries are waiting on. The output shows each in-flight query and its target. During a cascade, most queries cluster against one or a few upstream nameservers.

Step 3: Test for DNSSEC.

The fastest DNSSEC diagnostic is the +cd (checking disabled) flag test:

# If +cd succeeds and plain dig fails, DNSSEC validation is the cause
dig @127.0.0.1 example.com A +cd      # bypasses validation
dig @127.0.0.1 example.com A           # normal validation

If the +cd query returns NOERROR and the plain query returns SERVFAIL, the problem is DNSSEC validation. Check the ValFail counters, system clock accuracy, and trust anchor state:

timedatectl status
chronyc tracking 2>/dev/null
rndc managed-keys status

DNSSEC validation depends on accurate time. RRSIG signatures have inception and expiration times, so clock drift can cause validation failures for otherwise valid signatures.

Step 4: Check zone-specific failures.

For zone-specific SERVFAIL, verify the zone loaded successfully:

rndc zonestatus example.com
named-checkzone example.com /var/named/example.com.zone

If the zone failed to load, rndc status still shows “running” because named starts successfully even when individual zones fail. Check logs for zone load errors, especially after any rndc reload.

For authoritative secondaries, check the SOA expire countdown. A secondary that has expired a zone returns SERVFAIL for that zone only:

rndc zonestatus example.com | grep -i "expire\|refresh\|serial"

Metrics and signals to monitor

Fixes

Recursive resolution cascade

When upstream nameservers are slow or unreachable, recursive queries pile up and fill the recursive-clients table. New queries get SERVFAIL regardless of which domain they target.

• Identify the problem upstream. Run rndc recursing | awk '{print $NF}' | sort | uniq -c | sort -rn | head -10 to see which upstream nameservers are consuming slots.
• Consider lowering resolver-query-timeout temporarily. The default is 10000ms . Lowering it makes BIND fail fast rather than holding slots for the full timeout duration. This is a tradeoff: complex delegation chains that genuinely need more time will fail prematurely.
• Flush stuck cache entries if compounding. Use rndc flushname <domain> to clear negative cache entries for the affected domain. Do not use bare rndc flush unless you intend to clear the entire resolver cache, which increases upstream load while it repopulates.
• Raise recursive-clients if genuinely undersized. The default 1000 is too low for busy resolvers, but raising it requires sufficient file descriptors and memory. Setting it too high without those resources causes FD or memory exhaustion instead.

DNSSEC validation failure

• Verify system clock. Run timedatectl status and chronyc tracking. Even small drift can break RRSIG validation.
• Check trust anchors. Run rndc managed-keys status to verify trust anchor state. A corrupted managed-keys database or a stale root KSK after rollover causes broad validation failure.
• Do not disable validation to mask the problem. If a specific upstream zone has a signing problem, your resolver is correctly rejecting broken signatures. Disabling dnssec-validation accepts forged responses and removes a critical security control.
• Check version-specific requirements. On BIND 9.20.0 and later, dnssec-validation yes may require an explicitly configured trust-anchors statement. If you relied on implicit behavior in older versions, use dnssec-validation auto to let BIND manage trust anchors via RFC 5011.

Zone load failure

• Validate before reloading. Run named-checkconf /etc/named.conf and named-checkzone <zone> <zonefile> before every rndc reload.
• Check logs for the specific error. Run journalctl -u named --since "5 min ago" | grep -i "zone.*failed".
• Recover from journal corruption. Run rndc sync -clean followed by rndc reload for the affected zone. Journal corruption can prevent zone load even when the zone file itself is valid.
• Reload only the failed zone. Run rndc reload <zone> to avoid disrupting zones that loaded successfully.

Zone transfer expiry (authoritative secondaries)

• Check expire runway. Run rndc zonestatus <zone> on the secondary. Once the SOA expire timer runs out, the secondary stops serving the zone and returns SERVFAIL.
• Verify primary is reachable. Run dig @primary-ip <zone> SOA to confirm the primary is responding.
• Force transfer. Run rndc retransfer <zone> to initiate a full AXFR immediately.

Prevention

• Monitor SERVFAIL as a ratio, not a raw count. The ratio normalizes across traffic volume. Alert above 0.1%, escalate above 1%.
• Track RecursClients as a percentage of limit. This is BIND circuit breaker. When it trips, every recursive query fails. Daily peak should stay below 50% of the configured limit.
• Always validate before reload. Run named-checkconf and named-checkzone before every rndc reload. Verify rndc zonestatus after every reload.
• Monitor DNSSEC validation health. Track ValFail counters and system clock offset. Clock drift is a silent DNSSEC killer.
• Account for servfail-ttl in investigations. Negatively cached SERVFAIL responses persist for servfail-ttl after the upstream recovers. A SERVFAIL rate that does not drop immediately after a fix is expected behavior, not evidence the fix failed.
• Separate authoritative from recursive signals. In mixed-role deployments, aggregate statistics mask zone-specific authoritative failures behind recursive noise.

How Netdata helps

Netdata collects BIND statistics channel counters at per-second resolution, which makes SERVFAIL diagnosis faster by surfacing the correlations that distinguish root causes:

• QrySERVFAIL rate is collected as a derived metric, not just a cumulative counter, so the rate of change is visible immediately without manual delta computation.
• RecursClients is tracked as a gauge, showing utilization against the configured recursive-clients limit. The circuit breaker threshold is visible at a glance.
• DNSSEC validation counters (ValAttempt, ValOk, ValFail) are collected per-view, so you can see whether validation failures are broad or isolated to specific views.
• Resolver failure counters (QueryTimeout, Lame, QuerySockFail) are collected per-view alongside RecursClients, so the correlation between upstream timeouts and recursive client pressure is immediate.
• Cache hit ratio is derived from CacheHits and CacheMisses per-view, showing the early warning signal before a cascade develops into SERVFAIL.
• ML anomaly detection on these counters can flag a rising SERVFAIL trend or a RecursClients spike before it crosses a static threshold, giving lead time before the circuit breaker trips.

Related guides

• BIND REFUSED responses: ACL denials, recursion policy, and clients that get locked out
• named not responding on port 53: total outage versus UDP-works-TCP-fails
• BIND NXDOMAIN spike: DGA malware, water torture, and Windows suffix search lists
• rndc not responding: control-plane failure while queries still work
• BIND monitoring checklist: the signals every production resolver and authoritative server needs
• How BIND actually works in production: a mental model for operators
• BIND monitoring maturity model: from survival to expert
• BIND ’no more recursive clients: quota reached’: the recursive-clients circuit breaker
• BIND RecursClients climbing toward the limit: reading the recursive saturation gauge
• BIND recursive resolution cascade: one slow upstream taking down all resolution
• BIND forwarding loops: recursion that never terminates and burns recursive slots
• BIND resolver NumFetch per view: per-view recursive pressure in split-horizon setups