CoreDNS upstream connection cache misses: new connections adding latency per query

You are here because forwarded DNS queries got slower, or because coredns_proxy_conn_cache_misses_total is climbing and you need to know whether it matters. It does. Every connection cache miss means CoreDNS opens a fresh connection to an upstream resolver instead of reusing one from its pool. Setup cost lands on that query, and each new connection holds a file descriptor until it is closed or reaped.

This is usually degradation, not outage. CoreDNS keeps answering. The damage shows up as worse tail latency on uncached lookups, rising process_open_fds, and, under sustained churn, resource pressure that can turn into a real incident. Rule of thumb: a miss ratio above 50% sustained for 10 minutes is worth investigating.

The relevant counters are coredns_proxy_conn_cache_hits_total and coredns_proxy_conn_cache_misses_total, in the proxy subsystem, with labels proxy_name (typically forward), to (upstream address), and proto (udp, tcp, or tcp-tls). Those labels are what make the signal actionable: they tell you which upstream and which protocol is churning.

What this means

The forward plugin keeps a persistent connection pool per upstream. When a query needs forwarding, CoreDNS checks the pool for a reusable connection to that upstream. A hit reuses it. A miss opens a new one. For UDP the “connection” is lightweight. For TCP and DNS-over-TLS (tcp-tls) a miss means a handshake, and TLS adds negotiation on top. That setup cost is paid by the query that caused the miss.

Two forward plugin knobs govern pool behavior. expire controls how long an idle cached connection is kept before cleanup closes it; the documented default is 10 seconds. max_idle_conns caps idle connections per upstream per protocol; the default is 0, meaning unlimited. If you set neither, connections live for about 10 idle seconds and the idle pool has no explicit upper bound.

flowchart LR
  Q[Forwarded query] --> C{Connection in pool?}
  C -->|hit| R[Reuse connection
no setup cost]
  C -->|miss| N[New connection
handshake adds latency
consumes one FD]
  R --> U[Upstream resolver]
  N --> U
  U --> RC[Connection returned to pool
closed after expire
or by upstream]

A rising miss ratio has four plausible cause families: the upstream closes connections before CoreDNS can reuse them, CoreDNS discards connections too aggressively, the network path drops connections, or traffic is too sparse to keep connections warm. The job is to separate those before tuning anything.

Common causes

If misses are paired with intermittent forward errors on cached connections, check the changelog for your exact CoreDNS build for forward/proxy connection-cache fixes before tuning: the fix for the “cached connection was closed by peer” (ErrCachedClosed) comparison landed in CoreDNS 1.12.3 (PR #7427), so 1.12.2 and earlier builds lack it. Do not tune around a known connection-handling bug.

Quick checks

All of these are read-only. They assume the metrics endpoint is localhost:9153; adjust if yours differs.

# Connection cache hit and miss counters, per upstream and protocol
curl -s http://localhost:9153/metrics | grep 'coredns_proxy_conn_cache'

Look at the raw distribution first. If misses cluster on one to value, you have an upstream-specific problem. If they spread evenly with a high ratio everywhere, suspect configuration or traffic shape.

# Miss ratio over a 10-minute window, per upstream and protocol
sum by (to, proto) (rate(coredns_proxy_conn_cache_misses_total{proxy_name="forward"}[10m]))
  /
(
  sum by (to, proto) (rate(coredns_proxy_conn_cache_hits_total{proxy_name="forward"}[10m]))
  +
  sum by (to, proto) (rate(coredns_proxy_conn_cache_misses_total{proxy_name="forward"}[10m]))
)

Sustained values above 0.5 are the investigation trigger.

# File descriptor pressure: each new connection holds an FD until reaped
curl -s http://localhost:9153/metrics | grep -E '^process_(open|max)_fds'

# Per-upstream latency: misses should show up as a latency premium on forwarded queries
curl -s http://localhost:9153/metrics | grep 'coredns_proxy_request_duration_seconds'

# Upstream health: churn caused by a flapping upstream shows here
curl -s http://localhost:9153/metrics | grep 'coredns_proxy_healthcheck_failures_total'

# Confirm the upstream actually accepts and holds a TCP connection
dig @<upstream-ip> example.com +tcp +time=2 +tries=1

How to diagnose it

1. Quantify the miss ratio per upstream and protocol. Use the PromQL above. If nothing exceeds 50% sustained, this is informational, not an incident. Record the baseline and move on.
2. Isolate the protocol. UDP setup is cheap, so a high UDP miss ratio mostly signals churn without much per-query cost. TCP and TLS misses are where the latency premium lives. Focus there.
3. Isolate the upstream. If one to label accounts for most misses, test that upstream directly with dig +tcp and repeated queries. An upstream that closes idle connections faster than CoreDNS’s expire window will defeat the cache no matter how you tune CoreDNS. Verify the documented idle connection limits for your specific upstream (cloud VPC DNS, dnsmasq, etc.).
4. Check for upstream instability. Correlate miss bursts with coredns_proxy_healthcheck_failures_total{to=...} and the to-labelled latency histogram. A flapping upstream produces both health check failures and connection churn.
5. Check the FD trajectory. If process_open_fds trends upward with miss rate and never recovers, new connections are accumulating faster than cleanup reaps them. With max_idle_conns at 0 (unlimited), heavy churn can build real FD pressure. Compare against process_max_fds; above 80% utilization is warning territory.
6. Check traffic shape. At low per-upstream query rates, connections idle past expire between uses and every query is effectively a miss. That is expected behavior, not a fault. Plot miss ratio against forward query rate; if the ratio collapses as rate rises, your pool is fine and your traffic is just sparse.
7. Rule out version bugs. If misses come with forward errors on cached connections, confirm whether your CoreDNS build includes relevant forward/proxy pool fixes before changing expire or max_idle_conns.

Metrics and signals to monitor

Fixes

Upstream closes connections aggressively

If the upstream’s idle timeout is shorter than CoreDNS’s expire, connections die before reuse. Two options: shorten CoreDNS’s expire so CoreDNS stops handing out doomed connections (this does not reduce misses, but it avoids paying for a failed reuse attempt first), or switch affected traffic to UDP with prefer_udp if the upstream supports it and truncation/fallback behavior is acceptable, since UDP has no handshake cost. If the upstream is a managed resolver with documented connection limits, respect them rather than fighting them.

expire misaligned with traffic

If queries to each upstream arrive at intervals longer than expire, every query pays setup. Raising expire keeps idle connections longer and raises the hit ratio, at the cost of holding more FDs longer. That tradeoff is only safe if you also set max_idle_conns to bound the pool. Never raise expire while max_idle_conns remains 0 on a high-churn upstream; that is how a latency problem becomes FD exhaustion.

Unbounded pool under load

With max_idle_conns at its default of 0, bursts of misses create connections en masse and old ones take time to clean up. Set an explicit max_idle_conns per upstream sized to steady concurrency, then watch process_open_fds after the change to confirm the pool stays bounded.

Network instability

If misses correlate with latency jitter and health check failures across multiple upstreams, the path is the problem, not the pool. Tuning CoreDNS will not help. Work the network layer: routing, firewall connection tracking, and any middlebox that might reset idle TCP flows. Verify middlebox idle-TCP timeout behavior in your environment.

Sparse traffic

If the miss ratio is high only because per-upstream query rates are low, there is nothing to fix. Check whether the cache plugin is absorbing repeated names so connection reuse is irrelevant for most queries. A high DNS response cache hit ratio makes a high connection cache miss ratio mostly harmless.

Prevention

• Baseline the ratio per upstream. Miss ratio only means something against your own traffic pattern. Alert on deviation from baseline, with the 50%-over-10-minutes rule as the outer bound.
• Set max_idle_conns explicitly. The unlimited default is a latent FD exhaustion risk under any churn condition. Bound it, then monitor process_open_fds against process_max_fds.
• Tune expire with max_idle_conns as a pair. Longer retention plus a bounded pool is the safe combination. Longer retention with an unbounded pool is not.
• Track CoreDNS upgrades. Connection pool behavior has received bug fixes and rework across releases. Stale builds carry stale connection-handling bugs.
• Correlate, do not isolate. A miss ratio alert without upstream latency and FD context produces false confidence in both directions. Dashboard the three together.

How Netdata helps

• Netdata collects the CoreDNS Prometheus endpoint, so coredns_proxy_conn_cache_hits_total and coredns_proxy_conn_cache_misses_total are charted per to and proto without hand-built scrapes, giving you the per-upstream breakdown this diagnosis depends on.
• Per-second granularity on coredns_proxy_request_duration_seconds lets you see the latency premium from connection setup as it happens, rather than averaged away over a minute.
• process_open_fds is collected alongside CoreDNS metrics, so the “misses driving FD growth” correlation is visible on one dashboard instead of two tools.
• Health check failure metrics per upstream sit next to the connection cache charts, which makes the flapping-upstream versus aggressive-upstream distinction a visual comparison.
• Anomaly detection on the miss ratio surfaces drift from your baseline traffic shape, which is the alert you actually want for this signal.

Related guides

• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• How CoreDNS actually works in production: the plugin chain mental model
• CoreDNS monitoring checklist: the signals every production resolver needs
• CoreDNS monitoring maturity model: from survival to expert
• CoreDNS NOERROR with zero answers: the resolution failure that reports success
• CoreDNS NXDOMAIN vs SERVFAIL: why alerting on the wrong one buries real incidents
• CoreDNS per-upstream health check failures: degraded redundancy before total loss
• CoreDNS query rate dropped to zero while the process looks healthy
• CoreDNS returning REFUSED: no matching zone, an ACL, or the forward concurrency limit
• CoreDNS returning SERVFAIL: the resolver is failing queries and what to check first