Your resolver is returning SERVFAIL for major signed domains. Logs show broken trust chain resolving 'example.com' across dozens of unrelated domains. The per-view ValFail counter is climbing. Unsigned domains resolve normally, and authoritative zones you serve locally still answer.

This is a DNSSEC validation failure on the resolver side. BIND is rejecting signed answers because something in the chain of trust is broken locally. When the breakage is local (clock drift, stale trust anchors, corrupted managed-keys), every signed domain fails simultaneously. When it is upstream (an individual zone with expired signatures), only that zone is affected.

Diagnostic gate: if dig @127.0.0.1 <domain> +cd returns NOERROR but dig @127.0.0.1 <domain> returns SERVFAIL, the problem is DNSSEC validation, not recursion or upstream reachability. The +cd flag sets the CD (Checking Disabled) bit, telling BIND to skip validation and return whatever the upstream sent.

Broad ValFail across unrelated domains almost always means a local problem.

What this means

DNSSEC validation verifies a chain of cryptographic signatures (RRSIG records) from the root zone trust anchor down through TLDs to the final domain. If any link in that chain is broken, expired, or unverifiable, BIND rejects the answer and returns SERVFAIL.

The chain depends on two local conditions:

  • Accurate system time: RRSIG records carry inception and expiration timestamps. If the resolver clock is outside the validity window, every signature looks expired or not-yet-valid.
  • Valid trust anchors: BIND maintains the root zone KSK (Key Signing Key) in a managed-keys database, updated automatically via RFC 5011. If this database is stale, corrupted, or missing, BIND cannot establish the root of trust.

When either condition fails, every signed domain fails validation. Since major domains (.com, .org, google.com) are signed, the impact looks like a total DNS outage. It is not. Unsigned domains continue resolving because they have no signatures to validate.

There is no ValBadSig counter in BIND 9. Validation failures are aggregated into the single ValFail counter regardless of the specific cryptographic reason. Use the +cd test, clock checks, and managed-keys inspection to determine the root cause.

flowchart TD
    A["SERVFAIL for signed domains"] --> B{"dig +cd works?"}
    B -- "No" --> C["Not DNSSEC.
Check upstream or recursion"]
    B -- "Yes" --> D["DNSSEC validation confirmed"]
    D --> E{"Broad across
many domains?"}
    E -- "Yes, unrelated" --> F["Local problem"]
    F --> G{"Clock accurate?"}
    G -- "No" --> H["Fix NTP"]
    G -- "Yes" --> I["Check managed-keys"]
    I --> J{"Trust anchors OK?"}
    J -- "No" --> K["Repair managed-keys"]
    J -- "Yes" --> L["Check forwarder config"]
    E -- "No, one domain" --> M["Upstream signing issue"]

Common causes

CauseWhat it looks likeFirst thing to check
Clock drift (NTP failure)ValFail across all signed domains; RRSIG inception or expiration messages in logstimedatectl status and chronyc tracking
Stale or corrupt managed-keysValFail across all signed domains; may follow a root KSK rollover or BIND restartrndc managed-keys status
Forwarder without DNSSEC support“broken trust chain” when forwarding through an upstream that strips or corrupts DNSSEC responsesQuery the forwarder directly with dig +dnssec
Upstream zone signing failureValFail limited to one domain or TLD; all other signed domains validate normallydig +dnssec +cd @upstream-ns <domain> SOA

Quick checks

# Confirm DNSSEC is the cause: +cd disables validation
dig @127.0.0.1 example.com A +cd +time=2 +tries=1    # Works? Validation is the problem
dig @127.0.0.1 example.com A +time=2 +tries=1         # SERVFAIL? Confirms it

# Verify clock accuracy
timedatectl status
chronyc tracking

# Check trust anchor state (BIND 9.11.0+)
rndc managed-keys status

# Pull per-view DNSSEC validation counters from the statistics channel.
# Requires statistics-channels enabled in named.conf. Adjust the port to match.
# BIND's stats JSON uses arrays of {name, value} objects, not dicts.
<!-- TODO: verify field name is "value" vs "counter" across BIND 9.16/9.18/9.20 -->
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print('%s: %s=%s' % (v,s.get('name'),s.get('value'))) \
  for v,vd in d.get('views',{}).items() \
  for s in vd.get('resolver',{}).get('stats',[]) \
  if str(s.get('name','')).startswith('Val')]"

# Check overall SERVFAIL rate
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  print('QrySERVFAIL:', next((s.get('value') \
  for s in d.get('nsstats',[]) \
  if s.get('name')=='QrySERVFAIL'), 'N/A'))"

# Verify validation mode (auto is the default on modern BIND)
named-checkconf -p 2>/dev/null | grep dnssec-validation

# Test the intentionally-broken domain (should SERVFAIL if validation works)
dig @127.0.0.1 dnssec-failed.org A +time=2 +tries=1

# Check BIND uptime (cold starts produce transient validation noise)
rndc status | grep -i "uptime"

How to diagnose it

Step 1: Confirm DNSSEC is the cause. Run dig @127.0.0.1 <signed-domain> +cd and the same query without +cd. If the first returns NOERROR and the second returns SERVFAIL, validation is the problem. If both fail, look elsewhere: upstream reachability, recursive client exhaustion, or zone load failure.

Step 2: Determine breadth. Is ValFail climbing for many unrelated domains, or just one? Broad failure across unrelated domains points to a local problem. Isolated failure for a single domain or TLD points to an upstream signing issue you cannot fix locally.

Step 3: Check the clock. Run timedatectl status and chronyc tracking. DNSSEC depends on accurate time because RRSIG records carry inception and expiration timestamps. Even modest drift can cause validation failures if a signature is near its inception or expiration boundary. If the clock is wrong, fix NTP first. After correction, cached SERVFAIL responses may persist for the negative TTL. Use rndc flushname <domain> to clear cached failures for specific names, or rndc flush for the entire cache.

Step 4: Inspect trust anchors. Run rndc managed-keys status. This shows the state of each trust anchor, typically the root zone KSK. If the state is not “trusted” or the key ID does not match the current root KSK, the managed-keys database may be stale or corrupted. This can happen after a root KSK rollover if RFC 5011 automatic updates were disrupted, or if the managed-keys file was deleted or corrupted.

Step 5: Check forwarder configuration. If BIND forwards queries to an upstream resolver, that resolver must pass through DNSSEC records intact. If the forwarder strips RRSIG records or returns responses that do not match the DNSSEC chain, BIND logs “broken trust chain” and returns SERVFAIL.

Step 6: Rule out an isolated upstream issue. Query the upstream authoritative server directly: dig +dnssec +cd @<upstream-ns> <domain> SOA. If the RRSIG records are present and the inception or expiration window is valid, the upstream zone is fine and the problem is local. If the upstream signatures are genuinely expired, the issue is on their side and you can only wait or flush the cached failure.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
ValFail (per-view)Direct count of DNSSEC validation failures. Should be 0 or near-0.Sustained non-zero rate with ValAttempt above a floor
ValAttempt (per-view)Denominator for the ValFail ratio. If 0, the validator is inactive.Must be above a minimum floor before ValFail is meaningful
QrySERVFAILThe client-visible effect of validation failures.Rising rate correlated with ValFail
NTP clock offsetClock drift is the most common local cause of validation failure.Offset exceeding 30 seconds
named uptimeCold starts produce transient validation noise during key initialization.Uptime below 300 seconds can produce false ValFail
dnssec-validation configIf set to no, BIND accepts forged responses silently.Value is no or missing

Fixes

Fix clock drift

Correct the NTP daemon (chronyd or ntpd). Once the clock is accurate, flush cached SERVFAIL responses so clients do not continue seeing stale failures:

# Clear cached validation failures for a specific name
rndc flushname example.com

# Clear the entire cache. Disruptive: invalidates all cached entries,
# causes an immediate spike in upstream queries until the cache warms.
rndc flush

rndc flushtree may not clear BIND’s internal “bad cache” of validation failures. Use rndc flush or rndc flushname instead. If cached SERVFAIL responses persist after flushing, a named restart clears the bad cache.

Repair managed-keys

If rndc managed-keys status shows an incorrect or missing trust anchor, the managed-keys database needs reinitialization. This is a disruptive fix: BIND must restart to rebuild the trust anchor state from its built-in root KSK.

# Destructive: removes the trust anchor database. BIND will rebuild it
# from built-in root KSK on next start, but validation fails until restart completes.
# Path varies by distribution: /var/named/ on RHEL, /var/cache/bind/ on Debian.
<!-- TODO: verify exact managed-keys file path and name for your distribution -->
rm /var/named/managed-keys.bind
rm /var/named/managed-keys.bind.jnl

# Restart named to reinitialize trust anchors
systemctl restart named

On BIND 9.20+, the trusted-keys and managed-keys configuration statements were reportedly removed in favor of trust-anchors with initial-key or static-key. With dnssec-validation auto (the default), BIND manages trust anchors automatically and no manual trust-anchor statement is needed. If you use dnssec-validation yes on BIND 9.20+, you must provide an explicit trust-anchors statement.

Fix forwarder DNSSEC

If BIND forwards to an upstream that does not support DNSSEC, choose one:

  • Remove the forwarders statement so BIND queries authoritative servers directly and validates the full chain.
  • Ensure the upstream forwarder supports DNSSEC and passes RRSIG records through unchanged.
  • Set dnssec-validation no on the forwarding view as a temporary workaround.

The third option silently disables DNSSEC protection for all queries. It should not persist. Monitor the configuration to catch it.

Isolate upstream signing failures

If ValFail is limited to a single domain, the upstream zone has a signing problem you cannot fix locally. Use rndc flushname <domain> to stop caching the failure. Report the issue to the zone operator.

Prevention

  • Monitor NTP clock offset. Alert when drift exceeds 30 seconds. DNSSEC fails minutes to hours after NTP failure, and NTP is a critical dependency for any validating resolver.
  • Track ValFail as a ratio of ValAttempt. A small background rate from individual broken domains is normal on public resolvers. A sudden spike across many domains is not.
  • Gate alerts properly. Only alert on ValFail when recursion is enabled, ValAttempt is above a minimum floor (the validator is actively processing), and uptime exceeds 300 seconds. This filters cold-start noise and idle-server false positives.
  • Verify managed-keys after restarts. Managed-keys are validated on startup. Corruption may produce warnings but BIND still starts, with broken validation. Include rndc managed-keys status in your post-restart checklist.
  • Watch the dnssec-validation setting. Ensure it remains auto or yes with explicit trust anchors. A change to no silently disables security with no visible error.
  • Correlate ValFail with SERVFAIL rate. Validation failures manifest as SERVFAIL to clients. If SERVFAIL rises without a corresponding ValFail rise, the cause is not DNSSEC.

How Netdata helps

Netdata’s BIND collector surfaces the per-view DNSSEC validation counters (ValAttempt, ValOk, ValNegOk, ValFail) alongside QrySERVFAIL and recursive client utilization. Correlating these signals shortens diagnosis:

  • ValFail per second: reveals the exact moment validation started failing, which you can align with NTP status changes, BIND restarts, or configuration reloads.
  • ValFail to ValAttempt ratio: distinguishes a genuine validation storm from a low-traffic server where a handful of failures produce a misleading spike.
  • SERVFAIL correlation: confirms whether validation failures are actually reaching clients as SERVFAIL, or whether the cache is absorbing them.
  • NTP clock offset: Netdata’s system metrics collector tracks clock drift. Correlating a sudden offset change with a ValFail spike immediately identifies clock drift as the root cause.
  • Process uptime gate: filters cold-start false positives where managed-keys initialization produces transient validation noise during the first 300 seconds.
  • Per-view breakdown: in split-horizon deployments, identifies whether the failure is in one view or all views.