$ guides / bind-dns / bind-dns-query-logging-performance ▌

Operations Guides

BIND query logging in production: the gradual performance bottleneck teams forget to turn off

BIND query logging is a debugging tool, not a monitoring strategy. Teams enable it during an incident or security investigation, then forget to disable it. The result is a performance degradation so gradual that it gets attributed to traffic growth, hardware aging, or “BIND being slow.” By the time someone connects the dots, the resolver has been losing throughput for weeks.

With query logging enabled, BIND writes a formatted log line for every inbound query. A 50k QPS resolver can produce over 1GB per hour of log output. Each log entry adds I/O pressure that competes with query processing. There is no crash, no error message, no SERVFAIL spike. The resolver keeps answering queries, just more slowly, with throughput degrading over weeks or months. The degradation tracks traffic growth so closely that the two effects are indistinguishable without controlled measurement.

How query logging degrades performance

BIND’s per-query logging works through its internal logging framework. Worker threads queue a formatted log entry (timestamp, client IP, query name, query type, response code) for each inbound query. A dedicated logging thread drains that queue and writes to the configured destination.

At low query rates, the logging queue never fills and the cost is negligible. As volume grows, the queue fills faster than the logging thread can drain it. Worker threads that should be processing queries block waiting for queue space. The bottleneck shifts from DNS processing logic to I/O throughput.

Because the degradation is proportional to query rate, it tracks traffic growth. When traffic doubles over six months, the I/O pressure from query logging also doubles. Disabling query logging recovers the lost headroom immediately.

A related problem: rndc stats appends to its statistics dump file indefinitely and never overwrites. Monitoring scripts that call rndc stats on a cron job without truncating the file first will eventually fill the disk. The BIND ARM documents this: BIND always appends new statistics to the end of the file, so it grows continuously unless managed.

flowchart TD
    A["Query logging enabled"] --> B["Per-query log I/O"]
    B --> C["Logging queue fills"]
    C --> D["Worker threads block on queue"]
    D --> E["Query latency rises gradually"]
    E --> F["Throughput drops proportionally to QPS"]
    F --> G["Attributed to traffic growth"]
    G --> H["No investigation triggered"]
    H --> A

Common causes

Cause	What it looks like	First thing to check
Query logging left on after debugging	Gradual latency increase tracking traffic growth, disk I/O elevated on log partition	`rndc status` output for “query logging” state
Logging to syslog instead of file	Worse degradation than file logging, process-wide lock contention	`named-checkconf -p` for `category queries` channel destination
`rndc stats` file growing unbounded	Disk filling slowly, no apparent cause, statistics file is large	Check size of the statistics dump file
Query log file rotation not configured	Single log file growing to fill disk, or rotated but old files not pruned	Check log directory size and file count

Quick checks

# Check whether query logging is currently enabled
rndc status | grep -i "query logging"

# Check the logging configuration for query logging
named-checkconf -p /etc/named.conf 2>/dev/null | grep -A5 "category queries"

# Check the querylog option. If unset, query logging follows the presence
# of the logging category "queries".
named-checkconf -p /etc/named.conf 2>/dev/null | grep "querylog"

# Check query log file size and growth rate
ls -lh /var/log/named/queries.log 2>/dev/null || ls -lh /var/log/named/ 2>/dev/null

# Measure log write rate over 10 seconds
before=$(stat -c %s /var/log/named/queries.log 2>/dev/null)
sleep 10
after=$(stat -c %s /var/log/named/queries.log 2>/dev/null)
echo "Growth: $(( (after - before) / 10 )) bytes/sec, $(( (after - before) * 3600 / 10 / 1024 / 1024 )) MB/hour"

# Check rndc stats file size
ls -lh /var/named/data/named_stats.txt 2>/dev/null || ls -lh /var/cache/bind/named.stats 2>/dev/null

# Check disk I/O on the partition holding the logs
iostat -x 5 3 2>/dev/null || cat /proc/diskstats

# Check named process I/O wait
pidstat -d -p $(pgrep -x named) 5 3 2>/dev/null

# Check if logging goes to syslog
named-checkconf -p /etc/named.conf 2>/dev/null | grep -B2 -A2 "syslog"

How to diagnose it

Confirm query logging is enabled. Run rndc status and look for the query logging state. If it shows ON, you have found the problem. Also check the static configuration with named-checkconf -p | grep querylog, because query logging can be enabled in named.conf via the querylog yes; option or toggled at runtime with rndc querylog and persists only until restart.
Measure the log I/O volume. Use the growth-rate check above to estimate hourly log volume. On a 50k QPS resolver, expect multiple GB per hour. Even on a modest 5k QPS resolver, query logging produces hundreds of MB per hour. If the log file growth rate is proportional to your query rate, query logging is the source.
Correlate disk I/O with query latency. Use pidstat -d or iostat -x to measure disk I/O attributable to named. If disk write throughput is high and correlates with query rate, the I/O is from query logging. Cross-reference with external latency measurement: dig @127.0.0.1 example.com A response times should be under 5ms for cache hits. If cache-hit latency is creeping up, I/O contention is a likely cause.
Check for syslog as the logging destination. syslog is the worst possible destination for query logging. The syslog() call takes a process-wide mutex and performs a write for every log message. Under high QPS, this creates lock contention that affects all BIND worker threads, not just the one writing the log line. If category queries routes to a syslog channel, the performance impact is significantly worse than file-based logging.
Check the rndc stats dump file. If monitoring scripts call rndc stats periodically, the statistics file grows without bound. BIND appends to the file on every invocation. A script calling rndc stats every minute produces 1,440 appended blocks per day. Over weeks, the file can grow to fill the disk partition, especially if it shares space with zone files or journals.
Perform a controlled test. Disable query logging with rndc querylog and measure the immediate effect on throughput and latency. If you see a significant improvement, query logging was the bottleneck. Re-enable it with rndc querylog to confirm the degradation returns. This toggle is non-disruptive and takes effect immediately.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Query logging state (`rndc status`)	Directly tells you if per-query I/O is active	“query logging is ON” in production
Disk write rate on log partition	Quantifies the I/O cost of query logging	Write rate proportional to query rate, sustained
Query log file growth rate	Estimates disk fill timeline	Growth rate in hundreds of MB or GB per hour
`rndc stats` file size	Prevents unbounded growth filling disk	File growing without rotation or truncation
Cache-hit query latency	I/O contention shows up as processing delay	p50 latency for cache hits creeping above 5ms
CPU utilization vs query rate	Divergence indicates non-DNS overhead consuming CPU	CPU rising faster than query rate over weeks
Disk space on log partition	Prevents disk-full failures from log or stats files	Partition filling gradually with no apparent cause
Process I/O wait (`pidstat -d`)	Shows how much I/O the named process generates	Sustained high write KB/s for named

Fixes

Disable query logging immediately

The fastest fix is a runtime toggle that requires no restart:

# Toggle query logging off (if currently on)
rndc querylog
# Verify it is off
rndc status | grep -i "query logging"

This takes effect immediately. If query logging was enabled via rndc querylog at runtime (not in the config file), it will remain off until explicitly toggled again or until BIND restarts with a config that has querylog yes;.

To make the change permanent, ensure your named.conf does not have querylog yes; and does not route category queries to an active channel if you do not want query logging. If you want the category defined but logging off by default, use:

options {
    querylog no;
};
logging {
    category queries { queries_log; };
};

This lets you toggle query logging on temporarily with rndc querylog during investigations, while keeping it off by default.

Replace query visibility with dnstap

If you need per-query visibility for forensics or traffic analysis, dnstap is the recommended alternative. dnstap uses a structured binary format and writes to a Unix socket or file, with far lower overhead than text-based query logging. It has been available since BIND 9.11 and requires --enable-dnstap at compile time, plus the fstrm and protobuf-c libraries. Major distribution packages ship it compiled in: Debian and Ubuntu build with -Ddnstap=enabled, and Fedora/RHEL builds enable dnstap by default (%bcond_without DNSTAP in the bind spec), so the compile-time flag mainly matters for custom builds.

dnstap output can be decoded with the dnstap-read utility or consumed by an external process via the Unix socket. This decouples query capture from BIND’s query processing path, eliminating the I/O contention that makes text query logging so costly.

For continuous aggregate monitoring (query rates, response codes, cache hit ratio), use the statistics channel instead of either query logging or dnstap. The statistics channel must be explicitly configured in named.conf under statistics-channels {}. It provides JSON and XML endpoints that expose cumulative counters without per-query I/O:

# Poll statistics channel for query counters (no per-query I/O)
# Replace port 8653 with your configured statistics-channels port
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  ns=d.get('nsstats',{}); print('Requestv4:', ns.get('Requestv4',0), \
  'Requestv6:', ns.get('Requestv6',0))"

Fix rndc stats file growth

If you use rndc stats for monitoring, truncate or rotate the statistics file before each dump. This destroys previous entries in the file, but each rndc stats dump is a point-in-time snapshot, so historical blocks are rarely useful:

# Truncate before dumping (BIND appends, so the new dump is clean)
> /var/named/data/named_stats.txt
rndc stats
# Parse the most recent block only
tail -200 /var/named/data/named_stats.txt

Alternatively, switch to the statistics channel for programmatic monitoring. The statistics channel does not write to disk and provides the same counters in a parseable format. Polling every 5-10 seconds is sufficient for most monitoring needs. Avoid polling more frequently than every 5 seconds on busy resolvers, as the statistics channel query itself consumes resources.

Remove syslog as a logging destination

If query logging or any high-volume logging category routes through syslog, reconfigure it to use a file channel directly. syslog adds a process-wide mutex and write semantics that compound the I/O penalty. For BIND logging, file channels with versions and size directives for automatic rotation are the correct production pattern.

Prevention

Audit logging configuration as part of deployment. After any rndc reload or config change, verify that query logging is off. Add this to your deployment checklist.
Never enable query logging for continuous monitoring. Use the statistics channel for rates, response codes, and cache metrics. Reserve query logging for short investigative windows during incidents.
Monitor the query logging state. Include rndc status output parsing in your monitoring stack. Alert if query logging transitions to ON outside a planned investigation window.
Rotate or truncate the rndc stats file. If your monitoring calls rndc stats, truncate the file first. Better yet, migrate to the statistics channel.
Track disk I/O attributable to named. If write I/O grows proportionally with query rate over time, investigate whether query logging or another logging category is the source.
Use dnstap for forensic query capture. If you need per-query data for security analysis or traffic profiling, configure dnstap with a Unix socket consumer rather than enabling text query logging.

How Netdata helps

Per-second disk I/O metrics let you see write throughput on the log partition correlated with query rate. If disk writes track QPS tightly, query logging is the likely source.
CPU utilization trends plotted against incoming query rate reveal divergence. If CPU is rising faster than QPS over weeks, something other than DNS processing is consuming cycles.
Process-level I/O accounting (via eBPF or /proc metrics) shows how much read and write I/O the named process generates. Sustained high write rates for a DNS daemon are abnormal.
Disk space monitoring on the partition holding logs and statistics files catches the rndc stats growth problem before it fills the disk and causes a cascade of failures.
Query latency measurement via external probes, correlated with disk I/O and CPU, distinguishes between upstream-related latency (cache misses, slow authoritative servers) and local processing delays (I/O contention from logging).