rndc not responding: control-plane failure while queries still work

rndc status hangs. You Ctrl-C it, try again, same result. But dig @127.0.0.1 example.com A +short returns instantly with the right answer. The data plane is healthy. The control plane is dead.

You cannot flush caches, force zone transfers, dump state, reload configuration, or stop the daemon gracefully. If a cache-poisoning event or upstream degradation starts while the control plane is down, your primary incident-response tools are unavailable.

The BIND control channel runs over TCP port 953, separate from the port 53 query path. This separation is correct - a query flood should not saturate the management interface - but it means the control channel can fail independently while DNS resolution continues. The controls {} block in named.conf defines where named listens; rndc authenticates with a TSIG key over a short-lived TCP session.

Control-plane degradation often precedes a full query outage. If the root cause is thread starvation, file descriptor exhaustion, or memory pressure, the data plane will eventually fail too. A hung rndc is an early warning signal.

What this means

The key configuration files:

• rndc.conf: client configuration including TSIG key and target address/port. If this file exists, rndc uses it and ignores rndc.key.
• rndc.key: shared key file, typically generated by rndc-confgen -a. Used when no rndc.conf exists and no explicit controls {} statement references a different key.
• controls {} in named.conf: defines listening address, port, and allowed keys/clients. If absent, named defaults to listening on port 953 on loopback with the key from rndc.key.

A mismatch between the key in rndc.conf (or rndc.key) and the key in named.conf’s controls {} block causes authentication failure. Port 953 may be listening and accepting connections, but every command is rejected with “bad auth” in the logs.

If rndc status hangs rather than returning an error immediately, the problem is server-side: named accepted the TCP connection but cannot process the command because worker threads are saturated, file descriptors are exhausted, or internal event loops are stalled under load.

Common causes

Quick checks

Safe, read-only commands. Run them in order:

# Confirm the data plane is working
dig +time=2 +tries=1 @127.0.0.1 example.com A +short

# Test control-plane responsiveness with a hard timeout
timeout 5 rndc status 2>&1 || echo "rndc FAILED or TIMED OUT"

# Check if port 953 is listening
ss -ltnp '( sport = :953 )'

# Check active connections to the control channel
ss -tnp state established '( dport = :953 or sport = :953 )'

# Inspect the effective controls configuration
named-checkconf -p /etc/named.conf 2>&1 | grep -A10 "controls"

# Check file descriptor usage
CURRENT=$(ls /proc/$(pgrep -x named)/fd 2>/dev/null | wc -l)
MAX=$(grep "Max open files" /proc/$(pgrep -x named)/limits | awk '{print $4}')
echo "FDs: $CURRENT / $MAX"

# Look for authentication failures in BIND logs
grep -i "bad auth\|control\|rndc" /var/log/named/security.log 2>/dev/null | tail -20

# Check recursive client pressure (starvation indicator)
# Requires statistics-channels configured in named.conf
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  print('RecursClients:', d.get('nsstats',{}).get('RecursClients', 'N/A'))"

Log file paths (/var/log/named/security.log) and statistics channel port (8653) vary by deployment. Check your named.conf logging and statistics-channels configuration.

How to diagnose it

The approach depends on whether rndc returns an error quickly or hangs:

flowchart TD
    A[rndc status hangs or fails] --> B{Queries still resolve?}
    B -- No --> C[Full data-plane outage - check named process]
    B -- Yes --> D{Error or timeout?}
    D -- bad auth --> E[TSIG key mismatch]
    D -- connection refused --> F[Port 953 not listening or firewall]
    D -- Hangs over 5s --> G[Resource starvation: CPU, FD, threads]

Step 1: Classify the failure

Run timeout 5 rndc status 2>&1. If it returns within seconds with an error, the problem is likely configuration: key mismatch, missing listener, or firewall. If it times out after 5 seconds with no output, the problem is server-side resource starvation.

Step 2: If immediate error, check authentication

An immediate “bad auth” or “connection to remote host closed” error means the TCP connection succeeded but the TSIG key was rejected. Compare the keys:

# Show the key rndc will use
grep -A3 "key " /etc/rndc.conf 2>/dev/null || grep -A3 "key " /etc/rndc.key 2>/dev/null

# Show the key named expects in the controls block
named-checkconf -p /etc/named.conf 2>&1 | grep -A10 "controls"

# Look for the specific error in named logs
grep -i "bad auth\|invalid command" /var/log/named/security.log 2>/dev/null | tail -10

The secret string and key name must match exactly. If both rndc.conf and rndc.key exist, rndc uses rndc.conf and ignores rndc.key. Having both files with different keys is a common misconfiguration.

Step 3: If immediate error, check the listener

If the error is “connection refused”, verify port 953 is listening:

ss -ltnp '( sport = :953 )'
named-checkconf -p /etc/named.conf 2>&1 | grep -A10 "controls"

If there is no controls {} block, named should default to listening on 127.0.0.1 port 953 using rndc.key. If it is not listening, check named startup logs for errors.

On BIND 9.18 and later, configuring the control channel to use a Unix domain socket is a fatal error. In BIND 9.20.0, Unix domain socket support was removed entirely. If you upgraded from an older BIND version that used a Unix socket control channel, this will break rndc after the upgrade.

Step 4: If hang or timeout, check resource pressure

A hanging rndc status means named accepted the connection but cannot process the command. The most common cause is worker thread starvation under load:

# CPU utilization for named
pidstat -p $(pgrep -x named) 1 3

# Per-thread CPU distribution (check for single-thread bottleneck)
pidstat -t -p $(pgrep -x named) 1 3

# Recursive client count (each holds resources)
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  print('RecursClients:', d.get('nsstats',{}).get('RecursClients', 'N/A'))"

# File descriptor count
ls /proc/$(pgrep -x named)/fd | wc -l

If RecursClients is near the recursive-clients limit (default 1000, soft quota warning at 90%), or if FD usage is near the limit, the named process is under severe pressure. Worker threads are saturated by query processing, leaving control channel commands queued but unprocessed.

Step 5: Check for stuck connections

If rndc connections accumulate without completing, the control channel socket may be exhausted:

# Count established connections on port 953
ss -tn state established '( sport = :953 )' | wc -l

# Count all TCP states for port 953
ss -tan '( sport = :953 )' | awk '{print $1}' | sort | uniq -c

A large number of connections in ESTABLISHED or CLOSE_WAIT state on port 953 indicates connections that were opened but never properly closed.

Metrics and signals to monitor

Fixes

TSIG key mismatch

The fix is to align the keys. The simplest approach:

# WARNING: overwrites the existing rndc.key with a new key.
# named must be restarted afterward (rndc reload will not work -
# the control channel itself is broken). Plan a brief maintenance window.
rndc-confgen -a

If you use an explicit controls {} block with a custom key, ensure the key name, algorithm, and secret match exactly between rndc.conf (or rndc.key) and named.conf. After changing keys, restart named rather than relying on rndc reload (which requires a working control channel).

If both rndc.conf and rndc.key exist with different keys, remove one or align them. rndc.conf takes precedence if present.

Thread/resource starvation under load

This requires addressing the underlying load problem, not just the control channel:

1. Reduce recursive client pressure: If upstream nameservers are slow, lower resolver-query-timeout (default 10 seconds) to fail faster and free slots. Each timed-out query holds a slot for the duration of the timeout.
2. Increase FD limits: The default ulimit -n (often 1024) is too low for busy resolvers. Set LimitNOFILE in the systemd unit or configure OS-level limits. BIND’s files option is deprecated in 9.18 and removed in 9.20; use OS-level limits.
3. Scale recursive-clients: The default of 1000 may be too low for high-traffic resolvers, but increasing it without sufficient FDs and memory causes other exhaustion.

In BIND 9.20.0 and later, rndc -t <seconds> sets a client-side read timeout. This prevents rndc from hanging indefinitely but does not fix server-side starvation. It is a client-side mitigation, not a cure.

Port 953 not listening or firewall block

Verify the controls {} block in named.conf:

named-checkconf -p /etc/named.conf 2>&1 | grep -A10 "controls"

If the block is missing, add one. If it references a Unix domain socket, remove that configuration and use the default TCP port 953.

For firewall issues, ensure port 953 is accessible from localhost. Check iptables -L -n, firewall-cmd --list-ports, or cloud provider security group rules.

Chroot environments

If named runs in a chroot, rndc.key and rndc.conf must be visible inside the chroot directory, and the controls {} statement must reference the correct paths relative to the chroot root. A common failure mode after a BIND package update is the key file being updated outside the chroot while the copy inside remains stale.

Prevention

• Baseline rndc response time: Run timeout 5 rndc status periodically and alert if it fails or exceeds 1 second.
• Keep FD limits generous: Configure at least 65536 file descriptors for production named processes. Monitor FD usage as a percentage of the limit.
• Monitor RecursClients: Track recursive client count as a percentage of the recursive-clients limit. Alert above 50% sustained; page above 90%.
• Run named-checkconf before reloads: Catches syntax errors that could break the control channel.
• Use one key file: Standardize on either rndc.conf or rndc.key. Document which is authoritative for your deployment.
• Test after upgrades: BIND version changes, especially across major versions (9.16 to 9.18, or 9.18 to 9.20), can change control-channel behavior. Verify rndc status works after every upgrade.

Correlating with Netdata

Netdata provides several signals that help distinguish between rndc failure modes:

• Per-process CPU and thread metrics for named: Correlate control-plane hangs with single-thread saturation or aggregate CPU pressure. If CPU is low but rndc hangs, the cause is more likely key mismatch or socket exhaustion than thread starvation.
• File descriptor usage: Tracked as a percentage of the OS limit with per-second granularity. A rising FD trend coinciding with rndc degradation points to exhaustion.
• BIND collector metrics: RecursClients collected as an absolute gauge, viewable against the configured recursive-clients limit. Sustained high values explain why the control plane is starved.
• Memory and RSS tracking: Correlates with resource pressure that can degrade all named subsystems.

If rndc is slow and RecursClients is at 95% of limit with FDs near exhaustion, the root cause is resource starvation. If rndc returns an immediate error with normal resource metrics, the cause is key mismatch or a network/firewall issue.

Related guides

• How BIND actually works in production: a mental model for operators
• BIND monitoring checklist: the signals every production resolver and authoritative server needs
• BIND monitoring maturity model: from survival to expert
• BIND SERVFAIL responses: what a DNS SERVFAIL actually means and how to trace the cause