$ guides / coredns / coredns-too-many-open-files ▌

Operations Guides

CoreDNS too many open files: file descriptor exhaustion and refused connections

CoreDNS pods are logging too many open files and DNS resolution is failing intermittently or completely. Clients see timeouts or refused connections while the process keeps running and /health still returns 200. Restarting the pod fixes it for a while, then it comes back.

This is file descriptor exhaustion, and it is a cliff-edge failure. The moment process_open_fds reaches process_max_fds, CoreDNS cannot open anything new: no upstream connections, no listening sockets, no log files, no Kubernetes API watch streams. There is no queuing and no graceful degradation. New connection attempts fail immediately and DNS breaks.

The failure is visible well before the cliff if you watch the right signal, and the root cause almost always falls into a small set of categories.

What this means

Every file descriptor CoreDNS holds is an open resource: a socket to an upstream resolver, a listening socket on port 53, the persistent watch stream to the Kubernetes API server, a log file, the metrics endpoint. Each costs one FD. The process runs under a hard cap (process_max_fds, which reflects the effective ulimit applied by the OS, systemd, or the container runtime).

Two distinct situations produce exhaustion:

Legitimate growth. The workload genuinely needs more FDs than the limit allows. More upstream connections, more concurrent queries, more watch streams. The fix is raising the limit.
A leak. FDs are allocated and never released. The count rises steadily and never decreases, even when load drops. Raising the limit only delays the next incident. The fix is finding and stopping the leak.

Distinguishing these two is the core diagnostic task. A steadily rising process_open_fds that never returns to baseline is a leak. A count that tracks load and recovers after load drops is a capacity problem.

flowchart TD
  A[Upstream connections] --> E[process_open_fds]
  B[Listening sockets] --> E
  C[Kubernetes API watch streams] --> E
  D[Log files and misc] --> E
  E --> F{open_fds near max_fds?}
  F -->|no| G[Normal operation]
  F -->|yes, rising and never falling| H[Connection leak]
  F -->|yes, tracks load| I[Limit too low]
  H --> J[too many open files - DNS fails]
  I --> J

Common causes

Cause	What it looks like	First thing to check
Leaked upstream connections	`process_open_fds` climbs monotonically; high `coredns_proxy_conn_cache_misses_total`	Connection cache miss rate per upstream (`to` label)
Limit too low for the workload	FD count tracks query load; recovers when load drops	Current count vs `process_max_fds` during peak
Kubernetes API watch churn	FD count steps up after API connectivity issues; watch errors in logs	CoreDNS logs for watch disconnect/reconnect events
Slow upstream accumulation	FD growth correlates with rising goroutine count and upstream latency	`go_goroutines` and per-upstream latency
Misconfigured connection reuse	Cache miss ratio persistently high; new connection per query	`coredns_proxy_conn_cache_hits_total` vs `misses_total`

Quick checks

All read-only and safe to run during an incident.

# Current FD usage vs limit, from CoreDNS's own metrics
curl -s http://localhost:9153/metrics | grep -E '^process_(open|max)_fds'

# Same check from procfs (works even if metrics are unreachable)
ls /proc/$(pgrep coredns)/fd | wc -l
cat /proc/$(pgrep coredns)/limits | grep -i 'open files'

# What the FDs actually point at (sockets vs files)
ls -l /proc/$(pgrep coredns)/fd | awk '{print $NF}' | grep -c socket
ls -l /proc/$(pgrep coredns)/fd | awk '{print $NF}' | grep -vc socket

# Connection cache behavior - high misses mean new connections (new FDs)
curl -s http://localhost:9153/metrics | grep 'coredns_proxy_conn_cache'

# Goroutine accumulation - blocked upstream calls hold sockets open
curl -s http://localhost:9153/metrics | grep '^go_goroutines'

# REFUSED responses - clients being rejected
curl -s http://localhost:9153/metrics | grep 'coredns_dns_responses_total' | grep 'REFUSED'

# Recent CoreDNS logs for the telltale error
kubectl logs -n kube-system <coredns-pod> --tail=200 | grep -i 'too many open files'
kubectl logs -n kube-system <coredns-pod> --tail=200 | grep -iE 'watch|error'

The socket-vs-file split from /proc/<pid>/fd is the single most useful breakdown. If nearly all FDs are sockets, you are dealing with connection behavior (upstream pool or watch streams), not file handling.

How to diagnose it

Confirm exhaustion. Compare process_open_fds to process_max_fds. If open is at or near max, the mechanism is confirmed. Grep the logs for too many open files to confirm the symptom.
Classify the shape of the growth. Look at the process_open_fds trend over hours, not minutes. A monotonic climb that never decreases is a leak. A sawtooth that tracks query load is capacity. Step increases after specific events (API flaps, deployments, upstream changes) point at watch churn or connection pool misbehavior.
Identify what the FDs are. Use the socket-vs-file split from the quick checks. Sockets dominating means connections. In Kubernetes deployments, a persistent HTTP/2 watch stream to the API server is expected; many such streams is not.
Correlate with the connection cache. Check coredns_proxy_conn_cache_misses_total per upstream via the to label. A miss means a new connection was established, which means a new FD. If the miss rate is high or rising, connection reuse is failing: upstreams are closing connections aggressively, keepalive is misconfigured, or the pool is leaking. If hits dominate but the FD count still climbs, the pool is holding connections it never releases.
Check for blocked-call accumulation. If go_goroutines climbs alongside the FD count, queries are blocked on slow upstreams and each blocked query holds its connection open. This couples FD exhaustion to the slow-upstream failure mode: fix the upstream slowness and the FD pressure resolves. See CoreDNS goroutine count climbing: blocked upstream calls and leaks.
Check for watch churn. Look at CoreDNS logs for repeated watch disconnect and reconnect messages. Each reconnect cycle that fails to clean up the old stream can leak an FD. Correlate step increases in FD count with API server incidents.
Decide: leak or capacity. If the count recovers when load drops, raise the limit. If it never recovers, raising the limit buys time but you need the leak fixed.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`process_open_fds` / `process_max_fds`	Direct utilization against the cliff	Above 80% of max
`process_open_fds` trend	Distinguishes leak from capacity	Steady rise that never decreases
`coredns_proxy_conn_cache_misses_total`	Each miss is a new connection, a new FD	Miss ratio above 50% sustained
`coredns_proxy_conn_cache_hits_total`	Healthy reuse keeps FD count flat	Hit ratio dropping
`go_goroutines`	Blocked upstream calls hold sockets	Growth without matching QPS growth
`coredns_dns_responses_total{rcode="REFUSED"}`	Client-visible rejection during exhaustion	Any sustained nonzero rate
`coredns_forward_max_concurrent_rejects_total`	Rules in or out a different REFUSED cause	Nonzero means forward concurrency limit, not FDs

REFUSED responses during an FD incident look similar to REFUSED from max_concurrent rejects. Check coredns_forward_max_concurrent_rejects_total. If it is incrementing, you have a forward concurrency problem (see CoreDNS forward max_concurrent rejects), not FD exhaustion. If it is flat while REFUSED climbs and the FD count is at the ceiling, the FDs are the cause.

Fixes

Raise the file descriptor limit

The correct fix when the workload legitimately needs more FDs, and a reasonable stopgap while you hunt a leak.

Bare metal / systemd: set LimitNOFILE= in the service unit, run systemctl daemon-reload, then restart the unit. Restarting CoreDNS clears the cache, so expect a brief cold-cache latency bump and upstream load increase; stagger restarts across replicas.
Kubernetes: the container process inherits limits from the runtime and node configuration. Raise the FD limit at the container runtime or node level; pod-level ulimits are not universally supported across runtimes, so verify the effective value inside the running container with cat /proc/<pid>/limits after the change.

How high? Size from observed peak usage plus headroom, keeping at least 20% of the limit unused in normal operation. Do not set an absurd number to avoid thinking about it; an oversized limit hides leaks for months.

Fix connection reuse

If the connection cache miss ratio is high, FDs are being churned. Investigate why reuse is failing:

Upstreams closing idle connections aggressively.
Keepalive or connection-pool settings that force reconnection.
Network instability dropping connections between CoreDNS and upstreams.

Correlate misses by the to label to find which upstream is churning connections, and check that upstream’s latency and health with coredns_proxy_request_duration_seconds{to=...} and coredns_proxy_healthcheck_failures_total{to=...}.

Address the leak

If the FD count climbs monotonically and never recovers:

Correlate the climb with CoreDNS version changes and with the plugin mix in your Corefile. Connection and goroutine leaks have historically lived in specific plugins; upgrading to a release with leak fixes is often the actual fix.
If goroutines climb in lockstep, the leak is blocked calls, not the FD layer itself. Resolve the upstream slowness or timeout behavior first.
As an emergency measure only, restarting the pod resets the FD count. It also cold-starts the cache and re-establishes the API watch. Treat it as buying time, not a fix.

Prevention

Alert early, not at the cliff. Warn when process_open_fds exceeds 80% of process_max_fds, and add a trend alert on sustained growth without recovery. FD exhaustion gives you runway only if someone is watching the runway.
Trend the leak signature. A weekly review of the FD count minimum (the value during quiet hours) catches slow leaks that threshold alerts miss. If the quiet-hours floor keeps rising, something is leaking.
Watch connection reuse. Dashboard the conn-cache hit/miss ratio per upstream. Degrading reuse is the earliest indicator that FD pressure is coming.
Size the limit from measurement. Record peak FD usage during your highest-traffic periods and set the limit with comfortable headroom. Re-check after adding upstreams, zones, or replicas.
Correlate with deployments. Include FD count and goroutine count in your post-deploy checks for CoreDNS upgrades. Leaks introduced by a version change show up in the first hours.

How Netdata helps

Netdata surfaces the exact signals this article uses, on one timeline, which is what makes leak-versus-capacity classification fast:

process_open_fds and process_max_fds as a live utilization ratio, so you see the approach to the cliff at per-second resolution instead of after the logs fill with errors.
Connection cache hit/miss rates from coredns_proxy_conn_cache_*_total, broken down per upstream, so you can see reuse degrading before the FD count reacts.
go_goroutines next to FD count, making the blocked-call-accumulation pattern (and the slow-upstream coupling) visible at a glance.
Response codes including REFUSED, alongside coredns_forward_max_concurrent_rejects_total, so you can separate FD exhaustion from forward concurrency rejection without log spelunking.
Long retention on the FD floor, so the slow-leak signature (quiet-hours minimum creeping up over weeks) is visible in historical context.