$ guides / coredns / coredns-slow-upstream-latency ▌

Operations Guides

CoreDNS slow upstream: per-upstream latency, goroutine pileup, and the to label

Overall DNS latency is climbing. P99 is above 500ms, but P50 looks only mildly elevated. SERVFAIL is nonzero but nowhere near a full outage. The /health endpoint returns 200, every upstream passes its health checks, and yet clients are complaining that resolution is slow. Meanwhile go_goroutines is trending up and heap is following it.

This is the slow upstream drag pattern: one upstream DNS server is responding slowly but still returning valid answers. Because it never actually fails, the forward plugin’s health checks never mark it down, so queries keep getting routed to it. Each of those queries holds a goroutine while it waits. Latency accumulates, goroutines accumulate, and memory follows. Left alone, this ends in either max_concurrent rejects (if you set one) or OOM (if you did not).

The key diagnostic is the to label on coredns_proxy_request_duration_seconds. It breaks upstream latency down per upstream address, which turns “DNS is slow” into “upstream 10.0.0.53 is slow, the other two are fine.”

What this means

CoreDNS’s forward plugin sends uncached queries to the upstreams listed in your Corefile and load balances across them. Health checking decides which upstreams are eligible. The critical detail: the forward plugin’s health check treats any response that is not a network error as healthy. An upstream that answers every query correctly but takes 800ms to do it passes health checks indefinitely.

So the failure mode is not “upstream down.” It is “upstream degraded enough to be slow, not degraded enough to be removed.” Queries routed to the slow upstream sit in flight. CoreDNS gives each in-flight query a goroutine, and goroutine count is bounded only by memory. The pattern is self-reinforcing: high latency means queries stay in flight longer, which means more concurrent in-flight queries, which means more goroutines and more memory pressure.

flowchart TD
  A[Upstream responds slowly] --> B[Health checks still pass]
  B --> C[Queries keep routing to slow upstream]
  C --> D[In-flight queries accumulate]
  D --> E[Goroutines pile up]
  D --> F[P99 latency climbs]
  E --> G[Heap grows]
  G --> H{Bounded?}
  H -->|max_concurrent set| I[REFUSED rejects - backpressure]
  H -->|unbounded| J[OOM kill risk]

This is the mirror image of the upstream black hole pattern. In a black hole, all upstreams fail hard, health checks trip, queries fail fast with SERVFAIL, and latency is low. Here, latency is high and health checks are quiet. If your dashboards show high latency and you are waiting for a health check alert that never comes, this is why.

Common causes

Cause	What it looks like	First thing to check
Upstream overloaded	One `to` value shows elevated P99; upstream’s own metrics show load	`dig @<upstream> example.com +stats` from the CoreDNS pod
Network path degradation	Elevated and jittery latency to one upstream, others clean	Compare per-upstream latency via the `to` label
Cloud provider DNS rate limiting	Intermittent slowness and silent drops to a VPC resolver	Check per-upstream latency plus packet drop counters on the node
Connection cache misses	Latency elevated with rising `coredns_proxy_conn_cache_misses_total`	Check conn cache hit/miss ratio per upstream
GC pressure masquerading as upstream slowness	All upstreams look slow, `go_gc_duration_seconds` rising	Correlate `go_gc_duration_seconds` with latency spikes
Upstream slow only for some query types	Latency high but uneven across zones or record types	Break down by `rcode` and `zone` labels

The last two rows are the disambiguation: if every upstream is uniformly slow, the problem is probably inside CoreDNS (CPU saturation, GC, scheduling delay) rather than in any upstream. If exactly one upstream is slow, the problem is almost certainly that upstream or the path to it.

Quick checks

All of these are read-only.

# 1. Per-upstream latency (the money metric). Note the proxy subsystem and proxy_name label.
curl -s http://localhost:9153/metrics | grep 'coredns_proxy_request_duration_seconds'

# 2. Goroutine count. Baseline is typically 20-50; growth here is the pileup signal.
curl -s http://localhost:9153/metrics | grep '^go_goroutines'

# 3. Per-upstream health check failures. In this pattern, expect these to be flat.
curl -s http://localhost:9153/metrics | grep 'coredns_proxy_healthcheck_failures_total'

# 4. max_concurrent rejects, if you have the limit configured.
curl -s http://localhost:9153/metrics | grep 'coredns_forward_max_concurrent_rejects_total'

# 5. Overall request latency by zone, to see which traffic is affected.
curl -s http://localhost:9153/metrics | grep 'coredns_dns_request_duration_seconds'

# 6. Heap and GC, to assess how far the pileup has progressed.
curl -s http://localhost:9153/metrics | grep -E '^(go_memstats_heap_inuse_bytes|process_resident_memory_bytes)'
curl -s http://localhost:9153/metrics | grep 'go_gc_duration_seconds'

# 7. Test the suspected slow upstream directly from the CoreDNS pod.
dig @<upstream_ip> example.com +stats +time=2 +tries=1
# Compare "Query time:" against the healthy upstreams.

For check 1, the raw histogram output is verbose. What you want is a per-to comparison of the higher buckets or the _sum / _count ratio per to value. If you have Prometheus, histogram_quantile(0.99, sum by (to, le) (rate(coredns_proxy_request_duration_seconds_bucket{proxy_name="forward"}[5m]))) gives the per-upstream P99 directly.

Version note: before CoreDNS 1.11.0, this metric was coredns_forward_request_duration_seconds{to, rcode}. In 1.11.0 (PR #6178) the forward plugin metrics were restructured into the proxy subsystem with the proxy_name="forward" label. The old coredns_forward_* names are still emitted as deprecated, but new dashboards and alerts should use coredns_proxy_request_duration_seconds. If your metrics queries return nothing, check which naming your version emits.

How to diagnose it

Confirm the pattern shape. Check overall request latency (coredns_dns_request_duration_seconds): P99 high, P50 mildly elevated. Check SERVFAIL rate: nonzero but moderate, because some queries to the slow upstream eventually time out. Check coredns_proxy_healthcheck_failures_total: flat. High latency plus quiet health checks is the signature.
Isolate the slow upstream with the to label. Compare coredns_proxy_request_duration_seconds{proxy_name="forward"} per to value. You are looking for one upstream whose latency distribution is clearly worse than the rest. If all upstreams are equally slow, suspect CoreDNS itself (GC, CPU) or the shared network path instead.
Verify from the CoreDNS pod’s network position. Run dig @<slow_upstream> example.com +stats from the pod. This confirms the slowness is real at the network path CoreDNS uses, not a metric artifact. If dig is fast but CoreDNS-reported upstream latency is slow, the overhead is inside CoreDNS: check go_gc_duration_seconds and CPU throttling.
Measure the pileup. Look at go_goroutines over the last hour. A baseline of 20-50 that has climbed into the hundreds or thousands and is not coming back down is the accumulation. Then look at process_resident_memory_bytes against the container memory limit. RSS above 80% of the limit is your OOM runway alarm.
Check whether backpressure is configured. If your Corefile sets max_concurrent in the forward block, check coredns_forward_max_concurrent_rejects_total. Nonzero rejects mean the concurrency cap is absorbing the pileup and converting it into REFUSED responses. That is the limit working as designed, but it also means users are getting refused queries and the underlying slowness still needs fixing.
Check connection reuse. A rising miss rate on coredns_proxy_conn_cache_misses_total for the slow upstream adds connection setup time to every query. This can be a contributing factor or a symptom of the upstream dropping connections.
Decide: pull, replace, or ride it out. If one upstream is clearly the outlier and you have redundancy, the fix is usually to remove it. See the fixes section.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`coredns_proxy_request_duration_seconds` by `to`	Identifies which upstream is slow	One `to` value P99 above 250ms sustained while others are under 100ms
`go_goroutines`	Measures the blocked-query pileup	Sustained 2x baseline, or continuous growth without returning to baseline
`coredns_dns_request_duration_seconds` P99	User-visible impact	P99 above 500ms sustained; above 1s approaches client timeout territory
`process_resident_memory_bytes` vs container limit	OOM runway from goroutine and buffer growth	RSS above 80% of limit while goroutines grow
`coredns_forward_max_concurrent_rejects_total`	Shows backpressure firing if the cap is set	Any nonzero sustained rate
`coredns_dns_responses_total{rcode="SERVFAIL"}`	Some slow-upstream queries eventually time out	Moderate nonzero rate that tracks the slow upstream’s traffic share
`coredns_proxy_healthcheck_failures_total` by `to`	Confirms the upstream is NOT failing health checks	Flat is expected here; increments suggest the pattern is shifting to black hole
`go_gc_duration_seconds`	Rising heap from the pileup increases GC pressure	Pauses above 10ms adding to tail latency
Cache hit ratio	Slow upstreams can push queries past cache usefulness	Dropping ratio while upstream latency rises

Fixes

Remove or replace the slow upstream

The direct fix. Edit the Corefile’s forward line to drop the slow upstream (or swap in a replacement), then let the reload plugin pick it up or roll the pods. If you have multiple upstreams, the remaining ones absorb the traffic. Tradeoff: you are reducing redundancy while the bad upstream is out, so treat this as temporary and put a healthy upstream back.

Watch for the cache-flush side effect: a Corefile reload clears the cache, so expect a brief latency bump and a forward QPS spike right after the change as the cache warms.

Emergency: bound the damage with max_concurrent

If max_concurrent is not set, the pileup is unbounded and OOM is the end state. Setting a cap in the forward block converts unbounded goroutine growth into REFUSED responses once the cap is hit. REFUSED is a fast, visible failure that clients can retry against another resolver, which beats a silent OOM kill. Size it above your expected concurrent query load (query rate times upstream latency) with room for bursts. The CoreDNS documentation notes each concurrent query costs about 2KB of memory, which gives you an upper bound.

Tradeoff: once the cap is reached, excess queries fail. That is the point, but it means you must still fix the underlying slowness.

Emergency: restart as a last resort

If goroutines and RSS are already near the memory limit and a config change cannot land in time, restarting the pod clears the accumulated goroutines. This does not fix anything; the pileup will rebuild if the upstream is still slow. It buys time. Do not reach for this before removing the slow upstream, or you will be restarting on a loop. In Kubernetes, a restart also triggers the kubernetes plugin’s re-list, which is itself a memory spike, so make sure the memory limit has headroom for it.

Fix the upstream or the path

If the slow upstream is yours (an internal resolver, a DNS firewall, an inspection appliance), the CoreDNS-side changes are only mitigation. Investigate the upstream’s own load, the network path, and any cloud provider rate limits on the egress path.

Prevention

Dashboard the to label. Per-upstream latency is one of the most incident-critical and least dashboarded CoreDNS signals. Aggregate upstream metrics hide exactly this pattern.
Set max_concurrent. Unbounded concurrency turns a slow upstream into an OOM. A configured cap turns it into bounded, visible REFUSED pressure.
Alert on per-upstream P99 divergence, not just overall latency. One upstream at 3x the others is actionable hours before overall P99 crosses your page threshold.
Track goroutine trend, not just absolute count. A slow climb that never reverts is the early form of this pattern.
Keep memory headroom. RSS below 70% of the container limit gives the pileup somewhere to go while you respond, and leaves room for the restart re-list spike.
Run at least two upstreams so pulling a bad one does not leave you with a single point of failure.

How Netdata helps

Netdata charts coredns_proxy_request_duration_seconds with the to label preserved, so per-upstream latency divergence is visible without writing PromQL during an incident.
Goroutine count (go_goroutines), heap, and GC duration are charted alongside DNS latency on the same dashboard, which is exactly the correlation this pattern requires: latency up, goroutines up, health checks flat.
RSS relative to container limits surfaces OOM runway while the pileup is still growing, before the kernel makes the decision for you.
Anomaly detection on per-upstream latency flags the one drifting upstream even when aggregate latency is still inside normal range.
SERVFAIL and REFUSED response rates by plugin and RCODE let you distinguish slow-upstream timeouts from max_concurrent backpressure once a cap is in place.