BIND DNS Operations Guides on Netdata

BIND 'no more recursive clients: quota reached': the recursive-clients circuit breaker

Mon, 01 Jan 0001 00:00:00 +0000

The log message no more recursive clients (N/M): quota reached means BIND has run out of slots for in-flight recursive queries, and new queries are failing. Resolution for your clients is already degraded or broken.

The recursive-clients option (default 1000) caps the number of concurrent upstream fetches the resolver can have outstanding. A soft quota at 90 percent (default 900) starts shedding load before the hard limit. Once the hard limit is reached, every new recursive query returns SERVFAIL, including queries for domains whose upstream nameservers are healthy.

BIND 'too many open files': file descriptor exhaustion and silently dropped queries

Mon, 01 Jan 0001 00:00:00 +0000

BIND logs “too many open files” or “socket: file descriptor exceeds limit” and starts dropping queries. UDP health checks may still pass. Zone transfers fail intermittently. The rndc control channel becomes sluggish or unresponsive. Clients see random timeouts that look like upstream nameserver problems. This is file descriptor exhaustion, and the symptoms masquerade as network or disk issues.

The default ulimit -n of 1024 on many Linux distributions is low for a production DNS server. BIND consumes file descriptors (FDs) for every listener socket, outbound recursive query socket, TCP client connection, zone file, journal file, log file, and the rndc control channel. Under moderate load, a recursive resolver can exhaust 1024 FDs in minutes.

BIND cache eviction storms: DeleteLRU, an undersized max-cache-size, and the pressure spiral

Mon, 01 Jan 0001 00:00:00 +0000

DeleteLRU is rising fast while DeleteTTL barely moves. Cache hit ratio is dropping. Outbound recursive queries are climbing. RecursClients is trending toward its limit. This is a cache eviction storm: a self-reinforcing loop that can end in SERVFAIL for any query requiring recursion.

The cache is too small for the working set. BIND evicts entries by LRU before their TTLs expire. Each eviction forces a cache miss on the next query for that name, triggering an outbound recursive fetch. More fetches mean more concurrent recursive clients, higher latency per resolution, and less cache room as new entries from upstream compete with entries still under eviction pressure.

BIND cache hit ratio dropping: the leading edge of recursive pain

Mon, 01 Jan 0001 00:00:00 +0000

A dropping cache hit ratio is rarely the problem itself. It is the leading indicator that something is about to get worse. When fewer queries are answered from cache, each miss consumes a recursive-client slot, adds latency, and increases exposure to upstream slowness. On a busy resolver, a sustained hit-ratio decline from 95% to 80% can roughly triple outbound query volume and push recursive-client utilization into the danger zone.

The counter pair to watch is per-view CacheHits and CacheMisses in the BIND statistics channel cachestats. The ratio CacheHits / (CacheHits + CacheMisses) is what most monitoring systems alert on. Public recursive resolvers should sustain above 90% after warm-up. Alert with an uptime gate above 1800 seconds: cache warming takes 30 to 60 minutes after restart, and a cold cache naturally shows a near-zero hit ratio.

BIND clients-per-query and max-clients-per-query: duplicate recursion for popular names

Mon, 01 Jan 0001 00:00:00 +0000

When hundreds of clients query the same domain at the same instant, BIND does not send hundreds of identical recursive queries upstream. It sends one fetch and attaches the waiting clients to it. Two configuration knobs control how many waiters can attach before BIND starts dropping the overflow: clients-per-query (the soft limit, default 10) and max-clients-per-query (the hard ceiling, default 100).

How it works

BIND deduplicates concurrent identical queries at the fetch level. When the first client queries a name that requires recursion, BIND opens a single fetch to the upstream authoritative server. Subsequent clients querying the same name, type, and class while that fetch is in flight do not trigger additional upstream queries. They attach as waiters to the existing fetch. When the fetch completes, all attached waiters receive the answer.

BIND cold cache after restart: the warming storm and elevated upstream load

Mon, 01 Jan 0001 00:00:00 +0000

After a named restart, the recursive cache is empty. Every incoming query is a cache miss. On a resolver doing 50,000 qps, that means at least 50,000 outbound recursive queries per second to upstream authoritative servers – a 10x increase over steady-state outbound volume. This is the cache-warming storm, and it lasts 30 to 60 minutes.

Three things happen simultaneously: upstream load spikes, the recursive-clients table fills toward its limit, and cache hit ratio starts at zero and climbs. For the first 30 to 60 seconds, some queries may return SERVFAIL while root priming completes and authoritative zones finish loading. All of this is expected behavior.

BIND CPU saturation: single-core bottlenecks, DNSSEC crypto, and per-thread contention

Mon, 01 Jan 0001 00:00:00 +0000

BIND CPU saturation often hides from aggregate monitoring. Because named runs one worker thread per CPU core through netmgr, aggregate process CPU can read 25% on a 4-core host while a single core is pinned at 100%. Standard monitoring that checks total CPU utilization sees a healthy daemon. Clients see latency spikes and intermittent timeouts.

The core diagnostic challenge: BIND’s statistics channel reports no per-thread CPU data. You must measure at the OS level with mpstat, pidstat, or direct /proc inspection. Without per-core visibility, the symptom presents as unexplained latency with no obvious cause.

BIND DNSSEC failing from clock drift: NTP, RRSIG inception/expiry windows, and SERVFAIL

Mon, 01 Jan 0001 00:00:00 +0000

Signed domains start returning SERVFAIL while unsigned domains resolve normally. The named process is running, CPU is moderate, and the cache hit ratio has not collapsed. There was no recent rndc reload or configuration change. The failure appeared gradually over minutes or hours, not all at once.

This is the fingerprint of a DNSSEC validation failure caused by clock drift. Every RRSIG record carries an inception timestamp (when the signature becomes valid) and an expiry timestamp (when it stops being valid). BIND compares these against the local system clock. If the clock drifts far enough outside the RRSIG validity window, valid signatures appear expired or not-yet-valid, and BIND returns SERVFAIL for the entire domain.

BIND DNSSEC validation failing: 'broken trust chain', ValFail, and SERVFAIL for signed domains

Mon, 01 Jan 0001 00:00:00 +0000

Your resolver is returning SERVFAIL for major signed domains. Logs show broken trust chain resolving 'example.com' across dozens of unrelated domains. The per-view ValFail counter is climbing. Unsigned domains resolve normally, and authoritative zones you serve locally still answer.

This is a DNSSEC validation failure on the resolver side. BIND is rejecting signed answers because something in the chain of trust is broken locally. When the breakage is local (clock drift, stale trust anchors, corrupted managed-keys), every signed domain fails simultaneously. When it is upstream (an individual zone with expired signatures), only that zone is affected.

BIND dnssec-validation disabled: the security regression that 'fixes' SERVFAIL

Mon, 01 Jan 0001 00:00:00 +0000

SERVFAIL on signed domains is a real production problem. Users cannot reach major services because .com, .org, google.com, and countless other zones are DNSSEC-signed. When validation fails, those names stop resolving. The pressure to restore service is immediate.

The fastest way to make the SERVFAIL disappear is one line in named.conf:

dnssec-validation no;

After a reload, every signed domain resolves again. The monitoring dashboard goes green. The ticket closes. Everything works, including cache-poisoning attacks. The resolver now accepts forged responses for every query it processes.

BIND dynamic update failures: UpdateFail, denied updates, and TSIG drift

Mon, 01 Jan 0001 00:00:00 +0000

A rising UpdateFail counter on a BIND authoritative server means the dynamic update pipeline is broken. The counter does not tell you why. You need to correlate it with the security log, the zone serial, and the journal file state to narrow the cause.

Failures cluster into three families:

Authentication failures: TSIG key drift, wrong key names, clock skew past the 300-second fudge window.
Authorization failures: the zone’s allow-update or update-policy does not grant the presented key permission to modify the zone.
Journal and filesystem failures: disk exhaustion, permission errors, .jnl corruption from manual zone edits.

Each produces a distinct signal pattern in the logs and counters. Dynamic updates are denied by default. A zone must explicitly include allow-update or update-policy to accept updates. These two directives are mutually exclusive; configuring both is a configuration error and named will refuse to load the zone.

BIND forwarding loops: recursion that never terminates and burns recursive slots

Mon, 01 Jan 0001 00:00:00 +0000

Recursive-clients is climbing toward its limit. SERVFAIL responses are rising across unrelated domains. QueryTimeout counters are ticking up in the per-view resolver stats. It looks like a textbook recursive resolution cascade: an upstream nameserver is slow or unreachable, in-flight queries are piling up, and BIND’s circuit breaker is about to trip.

But when you run rndc recursing to identify the culprit, the upstream IP addresses are not external nameservers. They are your own infrastructure. Another BIND resolver you control, or this very server.

BIND inline signing silently failed: missing keys and a zone served unsigned

Mon, 01 Jan 0001 00:00:00 +0000

Your BIND authoritative server is running. Queries return answers. The zone loads without error. But validating resolvers worldwide are returning SERVFAIL for your domain, and there is no error line in your logs. The zone is configured for inline signing, but it is being served unsigned.

When a zone has inline-signing yes; with auto-dnssec maintain; or dnssec-policy default;, BIND should sign the zone transparently and maintain valid RRSIG records. If the signing key files are missing from the key-directory, BIND loads the unsigned zone and serves it as-is. No startup error. No log message. No statistics counter.

BIND journal (.jnl) corruption: dynamic-update and IXFR failures that block zone load

Mon, 01 Jan 0001 00:00:00 +0000

A zone fails to load after a restart or rndc reload. The named process is running, other zones respond normally, and named-checkzone reports the zone file is syntactically valid. The problem is the journal file (.jnl) alongside the zone file, which records pending dynamic updates and drives IXFR between primary and secondary servers.

Journal corruption has two operational faces. The acute case: named starts but refuses to serve the affected zone, logging “journal rollforward failed: journal out of sync with zone.” The subtle case: IXFR transfers keep failing and falling back to full AXFR. The zone stays current, but each refresh pulls the entire zone over TCP instead of just the diff, wasting bandwidth and CPU.

BIND lame delegations: 'lame server resolving' and nameservers that are not authoritative

Mon, 01 Jan 0001 00:00:00 +0000

The lame server resolving message in BIND’s lame-servers log category means the resolver contacted a delegated nameserver that answered but was not authoritative for the zone it was supposed to serve. Each lame encounter wastes a fetch cycle and adds latency. On busy resolvers with high query volume for affected zones, the cost compounds. The per-view Lame counter in resolver statistics tracks how often this happens.

Since CVE-2021-25219 , the lame-ttl default is 0, effectively disabling the lame cache. The option still exists in the configuration grammar, but with a zero default the cache does not retain lame indications. Before this change, the default was 600 seconds: BIND cached lame indications and skipped the offending server on subsequent queries within that window. Now every query for a zone with a lame delegation contacts that server from scratch, burning a full fetch cycle each time.

BIND managed-keys and trust anchors: KSK rollover, RFC 5011, and a stale root key

Mon, 01 Jan 0001 00:00:00 +0000

DNSSEC validation depends on a chain of trust anchored at the root zone’s Key Signing Key (KSK). BIND maintains that anchor automatically using RFC 5011 trust anchor management, stored in a managed-keys database. When it breaks, every signed domain on the internet fails validation simultaneously, and the resolver returns SERVFAIL for the majority of real-world queries.

The failure looks like total DNS failure because most major domains are signed. It is not a per-domain problem. One stale trust anchor breaks validation for every signed zone.

BIND max-cache-size: sizing the resolver cache without triggering the OOM killer

Mon, 01 Jan 0001 00:00:00 +0000

named disappears from the process table. dmesg shows the OOM killer selected it. systemd restarts it, the cache is cold, every query triggers recursion, upstream load spikes, and RSS climbs again. Within hours the cycle repeats. The root cause is often not a memory leak. It is the default max-cache-size.

Since BIND 9.11, the default max-cache-size for views with recursion yes is 90% of physical memory. On a dedicated resolver with 16 GB of RAM, that is 14.4 GB for the cache alone. On a shared box or a mixed-role server that also serves authoritative zones, that default guarantees the OOM killer will eventually visit.

BIND monitoring checklist: the signals every production resolver and authoritative server needs

Mon, 01 Jan 0001 00:00:00 +0000

BIND (named) processes queries through a pipeline: receive packet, parse, ACL/view check, cache lookup (if recursive), zone lookup (if authoritative), recursive fetch on cache miss, apply RPZ/DNSSEC, serialize response, send. Every signal below maps to a stage in that pipeline or a resource it competes for: CPU, memory, file descriptors, network buffers, source ports.

The levels are cumulative. Most signals come from the statistics channel (JSON at /json/v1/server, explicitly configured in named.conf). The rndc stats file is an alternative but appends indefinitely and can fill disk if not rotated. Commands below assume a single named process (standard deployment; BIND is multithreaded, not multiprocess).

BIND monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Most BIND deployments catch complete outages but miss the slow-burn failures that cause real incidents: cache pressure spirals, recursive client exhaustion, DNSSEC signature expiry, and kernel-level UDP drops that BIND itself never sees.

This article maps four monitoring maturity levels, from Survival to Expert. Each level adds signals that catch failure patterns invisible to the previous one. Use this as an inventory checklist: identify your current level, then decide which signals to add next based on whether you run a recursive resolver, an authoritative-only server, or a mixed-role deployment.

BIND named killed by the OOM killer: memory exhaustion and the cold-restart storm

Mon, 01 Jan 0001 00:00:00 +0000

named vanished from the process table. systemctl status named shows inactive (dead) or a restart timestamp from minutes ago. The likely cause: the Linux OOM killer terminated named when system memory was exhausted.

The cycle is self-reinforcing. BIND’s RSS grows steadily (excessive cache allocation, oversized RPZ, or a version-specific leak) until the kernel OOM killer selects named as the victim. All DNS resolution fails instantly. If systemd restarts the service, the cold cache forces every query through recursion, creating a warming storm that spikes upstream load. If the memory condition persists, the cycle repeats: start, grow, OOM, kill, restart.

BIND named RSS climbing: cache growth, allocator fragmentation, and real leaks

Mon, 01 Jan 0001 00:00:00 +0000

named RSS trending upward after warmup is the leading indicator before an OOM kill. The hard part is distinguishing three conditions that look identical on an RSS chart: cache-driven growth that will plateau under max-cache-size, allocator fragmentation that inflates RSS without a real leak, and genuine unbounded growth from a bug or misconfiguration.

RSS that climbs during a traffic burst or cache warmup and stays high is normal allocator behavior. BIND’s allocator (jemalloc on most builds) holds freed blocks for reuse rather than returning them to the OS via munmap. The operational question is not “is RSS high?” but “is RSS still growing, and how much runway remains?”

BIND NOTIFY not propagating: secondaries not refreshing when the primary changes

Mon, 01 Jan 0001 00:00:00 +0000

You updated a zone on the primary, reloaded with rndc reload, and confirmed the primary is serving the new serial. The secondaries still show the old one. Queries against them return stale data for minutes or hours. The zone is not broken, just delayed.

This is NOTIFY not reaching the secondaries. NOTIFY (RFC 1996) is a UDP push from the primary telling secondaries to check for a new SOA serial immediately, rather than waiting for the refresh timer. When NOTIFY is lost, blocked, or misconfigured, the secondary learns about changes only when its SOA refresh timer fires (commonly 1 hour for typical SOA values). The data looks delayed, not broken. The refresh timer eventually catches up, which is why this rarely pages, but it can mask more serious transfer problems if the refresh timer is the only thing keeping secondaries current.

BIND NXDOMAIN spike: DGA malware, water torture, and Windows suffix search lists

Mon, 01 Jan 0001 00:00:00 +0000

The QryNXDOMAIN counter jumping above 3x baseline is a common alert trigger, but the raw rate tells you almost nothing. Two resolvers with identical NXDOMAIN rates can be in completely different states: one healthy, one under active attack.

The signal that matters is query-name cardinality and entropy. If the same names repeat, the spike is benign (Windows suffix search lists, new client rollouts). If nearly every query name is unique and random, you are looking at a water torture attack or DGA malware beaconing. This distinction determines whether you page someone at 3 a.m. or close the alert.

BIND open recursive resolver: DNS amplification abuse and allow-recursion posture

Mon, 01 Jan 0001 00:00:00 +0000

An open recursive resolver answers recursive DNS queries from any source address. BIND configured with allow-recursion { any; } serves legitimate clients, but it also serves attackers: spoofed source IPs turn your resolver into a DDoS amplification relay, where small queries produce large responses directed at victims you have never heard of.

The abuse can be invisible. If the attack volume is small relative to your legitimate traffic, query rates look normal, cache hit ratios look healthy, and SERVFAIL rates stay flat. The only evidence is in the configuration itself and in subtle traffic pattern shifts: elevated ANY or TXT query shares, high source-IP diversity, or responses going to networks that have no business querying your resolver.

BIND query logging in production: the gradual performance bottleneck teams forget to turn off

Mon, 01 Jan 0001 00:00:00 +0000

BIND query logging is a debugging tool, not a monitoring strategy. Teams enable it during an incident or security investigation, then forget to disable it. The result is a performance degradation so gradual that it gets attributed to traffic growth, hardware aging, or “BIND being slow.” By the time someone connects the dots, the resolver has been losing throughput for weeks.

With query logging enabled, BIND writes a formatted log line for every inbound query. A 50k QPS resolver can produce over 1GB per hour of log output. Each log entry adds I/O pressure that competes with query processing. There is no crash, no error message, no SERVFAIL spike. The resolver keeps answering queries, just more slowly, with throughput degrading over weeks or months. The degradation tracks traffic growth so closely that the two effects are indistinguishable without controlled measurement.

BIND random subdomain (water torture) attack: an NXDOMAIN flood that bypasses the cache

Mon, 01 Jan 0001 00:00:00 +0000

Your BIND resolver is up, the process is running, UDP and TCP both answer on port 53. But clients across the network are experiencing slow DNS or outright SERVFAIL. NXDOMAIN rate has spiked to several times baseline. Cache hit ratio is collapsing. Recursive clients are climbing toward the hard limit. Upstream query rate has ballooned to approach or exceed the inbound rate.

This is a random subdomain attack, also called water torture or PRSD (Persistent Random Subdomain). Attackers flood your resolver with queries for randomized subdomains of a real domain: a1b2c3.victim.com, x7y8z9.victim.com, q2w3e4.victim.com. Every name is unique, so the cache cannot help. Each query is a miss, each miss forces recursion, and each recursion consumes a recursive-client slot and CPU. The victim’s authoritative server is hammered, but the real damage is collateral: your resolver becomes so saturated that resolution for every client, for every domain, degrades.

BIND RecursClients climbing toward the limit: reading the recursive saturation gauge

Mon, 01 Jan 0001 00:00:00 +0000

RecursClients is the number of queries currently in flight, each waiting for an upstream nameserver to respond. When this gauge climbs toward the configured recursive-clients limit, BIND is running out of slots to start new recursive lookups. Past the hard limit, every new recursive query receives SERVFAIL.

The metric is a gauge, not a rate. The absolute number means little without the configured ceiling next to it: 300 is comfortable against a limit of 1000 and dangerous against a limit of 350. Track RecursClients as a percentage of recursive-clients, not as a raw count, and watch the daily peak for runway estimation.

BIND recursive resolution cascade: one slow upstream taking down all resolution

Mon, 01 Jan 0001 00:00:00 +0000

SERVFAIL responses are flooding your recursive resolver. Users cannot resolve dozens of unrelated domains. But named is running, CPU looks moderate, and the authoritative zones on the same instance are still answering fine. This is the recursive resolution cascade: a single slow or unreachable upstream authoritative server consumes all available recursive-clients slots, and BIND returns SERVFAIL for queries that have nothing to do with the failing upstream.

Each in-flight recursive query targeting a slow upstream holds its slot for the full resolver-query-timeout duration (default 10 seconds). With retries across multiple nameservers, a single failed resolution can occupy a slot for 30 seconds or more. As stuck slots accumulate, capacity shrinks for unrelated queries. Once the hard limit is reached, every new recursive query receives SERVFAIL immediately. The signals that point to the real problem are in resolver internals that most teams do not monitor proactively.

BIND REFUSED responses: ACL denials, recursion policy, and clients that get locked out

Mon, 01 Jan 0001 00:00:00 +0000

REFUSED (DNS rcode 5) is a deliberate policy decision: BIND received the query, parsed it, and chose not to answer. The server is alive, listening, and processing queries. It is configured to reject this particular query from this particular source.

The actionable scenario: legitimate client subnets that previously resolved names or queried your zones suddenly start receiving REFUSED, typically after a configuration change. The fix is almost always an ACL or view mismatch, not a restart.

BIND resolver NumFetch per view: per-view recursive pressure in split-horizon setups

Mon, 01 Jan 0001 00:00:00 +0000

RecursClients is a global NSStats counter that reports the total number of recursive clients awaiting resolution across all views. It does not break down which view is consuming the pool. When RecursClients climbs to 847 out of 1000, you know the pool is stressed but not whether the pressure is evenly distributed or whether one view is responsible for most of it.

NumFetch per view fills that gap. It is a view-scoped gauge of active outbound fetches to upstream nameservers, maintained independently for each view in the resolver statistics. Unlike cumulative counters that increment over time, NumFetch is an instantaneous gauge: it goes up when a fetch starts and down when the fetch completes, times out, or is aborted. A sustained upward trend in one view’s NumFetch while other views remain flat localizes a view-specific resolution problem that the global counter masks.

BIND resolver RTT distribution shifting high: upstream nameserver degradation

Mon, 01 Jan 0001 00:00:00 +0000

The QryRTT histogram in BIND’s per-view resolver statistics is shifting toward higher millisecond buckets. What was once dominated by RTT10 and RTT100 (sub-100ms responses from upstream authoritative servers) is now accumulating in RTT500, RTT800, and RTT1600. In severe cases, most outbound recursive queries land in the overflow RTT1600+ bucket.

This metric is not client-perceived latency. The QryRTT counters measure the round-trip time of BIND’s outbound recursive queries to upstream authoritative nameservers. A rightward shift means the servers BIND depends on for cache misses are taking longer to respond. BIND has no native inbound client latency histogram; if you need end-to-end latency from the client perspective, use dnstap or external measurement.

BIND Response Rate Limiting (RRL): RateDropped, RateSlipped, and throttled legitimate clients

Mon, 01 Jan 0001 00:00:00 +0000

You see sustained non-zero RateDropped or RateSlipped counters in BIND’s statistics channel. Either RRL is absorbing a real DNS amplification or flood attack, or the configuration is too aggressive and silently dropping or truncating responses to legitimate clients. BIND’s RRL counters do not distinguish attacker from legitimate client. A dropped response is a dropped response, whether the source was a spoofed botnet node or a real resolver. Telling the difference requires correlating the counters with traffic patterns, source IP distribution, and TCP reachability.

BIND RPZ rewrites: threat-interception counters and reading a malware outbreak

Mon, 01 Jan 0001 00:00:00 +0000

Response Policy Zones (RPZ) let BIND rewrite DNS responses that match policy rules: blocked, dropped, redirected, or passed through with an exception marker. The RPZRewrites counter in BIND’s Name Server Statistics tracks how often this happens. When that counter spikes, it is usually the first telemetry signal that something inside your network is reaching known-bad infrastructure.

This article covers what RPZRewrites actually counts, what it silently excludes, how to distinguish a real malware outbreak from background noise, and what RPZ costs in query performance and memory. It assumes RPZ is already configured. For the broader BIND monitoring framework, see How BIND actually works in production: a mental model for operators.

BIND RRSIG expiry on authoritative zones: the silent signing time bomb

Mon, 01 Jan 0001 00:00:00 +0000

Your zone is down. Validating resolvers worldwide return SERVFAIL for every query to your domain. But the authoritative server looks fine: named is running, port 53 is listening, the statistics channel shows normal traffic. No errors in the logs. No alerts fired.

The problem is expired RRSIG signatures. BIND’s authoritative server does not validate its own signatures at serve time. It will serve expired RRSIGs indefinitely, and every validating resolver that queries your zone will reject the response. The outage is invisible from the server itself.

BIND secondary zone expired: the SOA expire timer runs out and the zone returns SERVFAIL

Mon, 01 Jan 0001 00:00:00 +0000

A single zone on your BIND secondary starts returning SERVFAIL. Other zones on the same server answer normally. The named process is up, CPU and memory look fine, and port 53 responds to health checks. Your monitoring shows green because it checks process liveness or queries a different zone. Only clients asking for that one zone are failing.

This is the end state of the zone staleness cascade. Transfers from the primary have been failing silently for days or weeks while the secondary kept serving the last-known-good data. The SOA expire timer counted down to zero, and BIND removed the zone from its database. The transition is binary: at one moment the zone works (stale but functional), at the next it is gone. Recovery requires a full AXFR, and that transfer must succeed or the cycle repeats.

BIND SERVFAIL responses: what a DNS SERVFAIL actually means and how to trace the cause

Mon, 01 Jan 0001 00:00:00 +0000

SERVFAIL (RCODE 2) means BIND attempted resolution and could not return a usable answer. It is not NXDOMAIN (name does not exist), REFUSED (policy denial), or FORMERR (malformed query). The process is alive, port 53 is open, the query was received, and the answer is failure.

This makes SERVFAIL invisible to binary health checks. A BIND server returning 100% SERVFAIL to every client still passes process-liveness and port-check probes. SERVFAIL is a symptom with many possible causes: upstream nameserver timeouts, DNSSEC validation failures, recursive-clients exhaustion, broken delegation, a zone that failed to load, or a configuration error. Tracing it requires correlating multiple BIND signals.

BIND SOA expire runway: the countdown that predicts a silent secondary-zone outage

Mon, 01 Jan 0001 00:00:00 +0000

A BIND secondary serving stale data is not broken. It answers queries with old but functional records, and from the outside it looks healthy. The danger is the countdown underneath: the SOA expire timer, ticking from the last successful zone transfer. When it reaches zero, the secondary stops serving the zone and returns SERVFAIL or REFUSED for every query against it.

Most teams monitor SOA serial consistency between primary and secondary. That catches “the secondary is behind” but not how much time remains before it gives up entirely. The expire runway does. It degrades over hours to days, making it ideal for early warning. It is the difference between catching a transfer failure with days of runway and discovering it after the zone has expired and clients are failing.

BIND SOA serial mismatch: a secondary serving stale data behind the primary

Mon, 01 Jan 0001 00:00:00 +0000

The secondary’s SOA serial is behind the primary. Clients querying the secondary get stale A values, missing new entries, deleted hosts still resolving. The server responds NOERROR because its zone data is internally consistent. It is not current.

This is the precursor to zone expiry, and it is silent. The secondary serves stale but functional answers for the entire SOA expire period, typically 1 to 4 weeks. Monitoring that checks “can I resolve this zone?” passes. Health checks on port 53 pass. Everything looks fine except the data is wrong. The only signal is the serial number gap between primary and secondary, and BIND’s statistics channel does not expose it. You must probe externally.

BIND tcp-clients exhaustion: the TCP connection limit, transfers, and truncation fallback

Mon, 01 Jan 0001 00:00:00 +0000

BIND is up. rndc status shows “running.” Your UDP health check returns NOERROR in under 2ms. But zone transfers are failing, some DNSSEC-validated queries time out, and clients receiving large responses report intermittent connection refused on TCP/53. If your monitoring only probes UDP, you will not know anything is wrong until a secondary’s zone expires or a downstream resolver escalates a ticket.

The likely cause is tcp-clients exhaustion. BIND caps concurrent inbound TCP connections at a configurable limit (default 150 in BIND 9.18 and 9.20). When that quota is full, new TCP connections to port 53 are refused. Zone transfers (AXFR/IXFR) cannot complete. DNSSEC answers that exceed the EDNS0 UDP buffer size are truncated, forcing clients to retry over TCP, and that retry fails too. The named process stays healthy and UDP resolution continues for everything that fits in a single UDP datagram.

BIND TSIG failure on zone transfer: BADKEY, BADTIME, and refused transfers

Mon, 01 Jan 0001 00:00:00 +0000

Zone transfers between your BIND primary and secondary have stopped. The secondary is falling behind on SOA serial, and the logs show TSIG verification failures: BADKEY, BADTIME, or BADSIG. The transfer is refused, and the secondary silently drifts toward zone expiry.

The failure hides in the security and xfer-in logging categories, not in the main query path. A secondary can serve stale data for days or weeks until the SOA expire timer runs out, at which point it stops serving the zone entirely and returns SERVFAIL. Nothing alerts until the zone disappears.

BIND UDP packet-rate saturation: softirq, single-core bottlenecks, and pps limits

Mon, 01 Jan 0001 00:00:00 +0000

DNS queries are timing out for some clients. BIND statistics look clean: query rate is steady, SERVFAIL is low, cache hit ratio is normal. The link is not saturated. BIND logs show nothing unusual. But clients keep reporting intermittent failures with no apparent correlation to domain, client subnet, or time of day.

If you have ruled out upstream issues, cache pressure, and DNSSEC failures, the problem may be in the kernel. When the host cannot process UDP packets fast enough, the kernel drops them before BIND reads them from the socket. BIND has no visibility into these drops: no log entry, no statistics counter, no error. The only evidence lives in kernel-level UDP counters that most DNS monitoring setups never collect.

BIND UDP receive buffer errors: the invisible query loss named never logs

Mon, 01 Jan 0001 00:00:00 +0000

Clients report intermittent DNS timeouts. Retries are up. Your monitoring says named is healthy: the process is running, port 53 is open, functional queries succeed, and BIND’s statistics channel shows reasonable response rates with no elevated SERVFAIL. You suspect upstream nameserver problems, network path issues, or client-side misconfiguration. None of those investigations turn up anything.

The problem may be happening between the kernel and BIND, in a layer where named has zero visibility. When the kernel’s UDP receive buffer overflows, packets are silently dropped before BIND ever reads them from the socket. There is no log entry, no statistics counter increment, no error of any kind inside named. The only evidence lives in kernel-level counters that most BIND monitoring setups never collect.

BIND UDP vs TCP query ratio: truncation, EDNS negotiation, and TCP fallback spikes

Mon, 01 Jan 0001 00:00:00 +0000

A rising TCP share of DNS queries is one of the most informative early warning signals in BIND. TCP normally carries under 5% of query volume on a recursive resolver. When that ratio shifts upward, something has changed in the resolution path: responses are being truncated, EDNS negotiation is failing, zone transfers are spiking, or a firewall is interfering with DNS traffic.

The shift matters more than the absolute count. A jump from 2% to 15% TCP share tells you the resolution path is under stress, regardless of total query volume.

BIND unauthorized zone transfer attempts: AXFR/IXFR from sources not on allow-transfer

Mon, 01 Jan 0001 00:00:00 +0000

Your BIND logs show denied AXFR or IXFR requests from unknown source IPs. The entries appear under the security category at error severity:

client 198.51.100.42#53124: zone transfer 'example.com/AXFR/IN' denied

Denied attempts mean your ACL is working. The critical question is whether any unauthorized transfer succeeded, because a successful AXFR exfiltrates the entire zone: every record, internal hostnames, SRV targets, TXT metadata, and the full infrastructure topology.

Repeated denied attempts from unauthorized sources are reconnaissance, which warrants a ticket. A confirmed successful unauthorized transfer means your zone data has been exfiltrated, which is a page.

BIND upstream query timeouts: QueryTimeout, retries, and the 30-second resolution stall

Mon, 01 Jan 0001 00:00:00 +0000

When a BIND resolver sends a recursive query upstream and gets no response, it waits. The default wait is 10 seconds, and BIND may retry the query up to 3 times before giving up. A single failed resolution can occupy a recursive-client slot for 30 seconds or more. If the upstream failure is broad enough, those slots fill, the recursive-clients limit is reached, and every new recursive query starts returning SERVFAIL.

BIND zone not loaded after reload: 'loading from master file failed' and the silent zone outage

Mon, 01 Jan 0001 00:00:00 +0000

You ran rndc reload after editing a zone file. rndc status reports the server is running. systemd says the service is active. Your monitoring confirms the process is alive and port 53 is open. But something is wrong with one specific zone.

named does not crash when a single zone fails to load. It logs the error and continues serving the zones that loaded successfully. Two outcomes are possible for the affected zone:

BIND zone transfer failed (AXFR/IXFR): reading xfer-in failures before the zone goes stale

Mon, 01 Jan 0001 00:00:00 +0000

A secondary BIND server’s zone transfers are failing. The zone is still being served, clients are still getting answers, and your health checks are still green. The secondary is serving stale data from the last successful transfer, and the SOA expire timer is counting down. When it reaches zero, the secondary stops serving the zone and returns SERVFAIL or REFUSED for every query.

Depending on the zone’s SOA values, you may have hours, days, or weeks of runway before the zone goes dark, with no visibility into the countdown unless you are specifically tracking it.

How BIND actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

During an incident, named can feel like a black box. A single process handles packet reception, DNS parsing, ACL evaluation, cache lookup, zone lookup, recursive fetch, DNSSEC validation, RPZ policy enforcement, and response serialization. There is no separate resolver process, no separate authoritative process, no separate cache daemon. Every query flows through the same pipeline, and every subsystem competes for the same pool of CPU, memory, file descriptors, and kernel network buffers.

named not responding on port 53: total outage versus UDP-works-TCP-fails

Mon, 01 Jan 0001 00:00:00 +0000

named can be alive (pgrep -x named returns a PID) yet not answering on the serving interface. Process liveness checks pass, but clients experience timeouts and retries that propagate to every service depending on DNS.

Two diagnostic forks determine the fix path. The first: total outage (both UDP and TCP fail on the serving interface) versus UDP-works-TCP-fails. A total outage means the main DNS path is broken. A UDP-works-TCP-fails condition leaves large responses, DNSSEC-heavy answers, and zone transfers at risk while UDP-only health checks stay green. The second fork: loopback success versus serving-interface success. A query that succeeds against 127.0.0.1 confirms the daemon processes queries, not that real clients can reach it.

rndc not responding: control-plane failure while queries still work

Mon, 01 Jan 0001 00:00:00 +0000

rndc status hangs. You Ctrl-C it, try again, same result. But dig @127.0.0.1 example.com A +short returns instantly with the right answer. The data plane is healthy. The control plane is dead.

You cannot flush caches, force zone transfers, dump state, reload configuration, or stop the daemon gracefully. If a cache-poisoning event or upstream degradation starts while the control plane is down, your primary incident-response tools are unavailable.

The BIND control channel runs over TCP port 953, separate from the port 53 query path. This separation is correct - a query flood should not saturate the management interface - but it means the control channel can fail independently while DNS resolution continues. The controls {} block in named.conf defines where named listens; rndc authenticates with a TSIG key over a short-lived TCP session.

Verifying rndc reload: catching a failed zone load before your clients do

Mon, 01 Jan 0001 00:00:00 +0000

rndc reload queues a zone reload and returns immediately with a success message. It does not confirm that the zone actually loaded. If the zone file has a syntax error, a missing include, or a permission problem, the error goes into the BIND log and nowhere else. The operator sees success. The zone is either stale (on reload failure, BIND continues serving the previous version) or answering SERVFAIL/REFUSED (on restart or initial load failure).