CoreDNS goroutine count climbing: blocked upstream calls and leaks

Your go_goroutines graph for CoreDNS is trending up. Maybe it spiked during an incident and never came back down. Maybe it has been climbing slowly for days. Either way, the question is the same: are queries piling up behind a slow upstream, or is something leaking goroutines that will never exit?

The distinction matters because the fixes are completely different. A blocked-upstream spike resolves when the upstream recovers. A leak grows until the OOM killer ends the debate for you.

What this means

CoreDNS handles each in-flight DNS query on its own goroutine, plus a small set of background goroutines for cache maintenance, Kubernetes API watches, health checks, and metrics. Go’s runtime spawns and reaps these cheaply, so the goroutine count is a live proxy for concurrent work.

A healthy instance sits at a baseline of roughly 20-50 goroutines. Under load the count tracks approximately with QPS multiplied by average query duration: fast queries mean goroutines are created and destroyed so quickly the count stays low. Two failure shapes break this pattern:

• Blocked upstream calls. A query that needs forwarding holds its goroutine until the upstream responds or the forward timeout fires. If an upstream gets slow but not dead, in-flight queries accumulate. Goroutines climb, latency climbs with them, and each goroutine holds memory. This pattern is self-limiting: when the upstream recovers, the backlog drains and the count returns to baseline.
• A goroutine leak. A goroutine blocked on something that will never complete: an unclosed connection, a deadlocked channel, a plugin bug. The count ratchets upward regardless of load and never returns to baseline. At roughly 4KB minimum stack per goroutine plus per-goroutine allocations, enough leaked goroutines push the process toward its memory limit and an OOM kill.

The operational dividing line: a sustained count above 2x your stable baseline for more than 10 minutes warrants investigation. Above 5000 goroutines combined with rising memory is a strong leak signal. A startup spike that settles within about 30 seconds is normal Go runtime behavior, not a leak.

Common causes

The version-specific leaks are worth naming because the fix is an upgrade, not a config change: the transfer plugin leaked a goroutine on AXFR errors before 1.12.4, and the secondary plugin leaked one on reload before 1.13.2. The DoQ server spawned an unbounded goroutine per QUIC stream before 1.12.2, and a regression (CVE-2026-32934) kept spawning unbounded waiter goroutines until 1.14.3. If you are on an affected version and the leak signature matches, upgrading is the fix.

Quick checks

These are all read-only and safe to run against a production instance.

# Current goroutine count
curl -s http://localhost:9153/metrics | grep '^go_goroutines'

# Per-upstream forward latency (which upstream is slow?)
curl -s http://localhost:9153/metrics | grep 'coredns_proxy_request_duration_seconds'

# Per-upstream health check failures
curl -s http://localhost:9153/metrics | grep 'coredns_proxy_healthcheck_failures_total'

# Forward concurrency backpressure (if max_concurrent is set)
curl -s http://localhost:9153/metrics | grep 'coredns_forward_max_concurrent_rejects_total'

# Heap and GC state
curl -s http://localhost:9153/metrics | grep -E '^(go_memstats_heap_inuse_bytes|go_gc_duration_seconds|process_resident_memory_bytes)'

# Open file descriptors (connection leak signature)
curl -s http://localhost:9153/metrics | grep '^process_open_fds'

# Query rate for proportionality check
curl -s http://localhost:9153/metrics | grep '^coredns_dns_requests_total'

Note on metric names: from CoreDNS 1.5.0 through 1.11.x the per-upstream metrics appear as coredns_forward_request_duration_seconds, coredns_forward_healthcheck_failures_total, and coredns_forward_conn_cache_misses_total; since CoreDNS 1.12.0 they are exported as coredns_proxy_request_duration_seconds{proxy_name="forward"}, coredns_proxy_healthcheck_failures_total{proxy_name="forward"}, and coredns_proxy_conn_cache_misses_total{proxy_name="forward"}. If a grep returns nothing, try the other name before concluding the signal is absent.

Also check the CoreDNS version against the leak fixes listed above, and look at the pod restart count in Kubernetes: repeated OOMKilled restarts with a climbing goroutine history before each kill is the leak archetype reaching its conclusion.

How to diagnose it

The core diagnostic is shape analysis: does the curve track load, or does it ratchet?

flowchart TD
  A[go_goroutines climbing] --> B{Tracks QPS and returns to baseline?}
  B -- Yes --> C[Blocked upstream calls]
  B -- No, ratchets up --> D[Goroutine leak]
  C --> E[Find slow upstream via per-upstream latency]
  E --> F[Remove or replace slow upstream]
  D --> G[Capture goroutine profile with pprof]
  G --> H{Stack trace points to a plugin?}
  H -- Yes --> I[Check version vs known leak fixes]
  H -- No --> J[Check DoQ enabled + version vs DoQ CVEs]
  I --> K[Upgrade CoreDNS]
  J --> K

1. Establish the baseline. Look at go_goroutines over the last 24-72 hours, not just the incident window. You need to know what “normal” is for this instance before judging the spike. Baselines of 20-50 are typical; yours may differ with heavy watches or DoQ enabled.

2. Test proportionality. Overlay the goroutine count against query rate (coredns_dns_requests_total) and P99 latency (coredns_dns_request_duration_seconds). If goroutines grew 10x while QPS grew 2x, queries are blocking, not just arriving. If goroutines grow with no QPS change at all, suspect a leak immediately.

3. Check the recovery behavior. This is the single most diagnostic observation: after the load or latency spike ends, does the count return to baseline? Blocked-call accumulation drains within minutes of the upstream recovering. A leak does not drain. Watch the 10-30 minutes after the event.

4. Identify the slow upstream. If it is blocked calls, break down coredns_proxy_request_duration_seconds by the to label. One upstream at 500ms+ while others are at 20ms tells you exactly where the backlog is forming. Cross-check coredns_proxy_healthcheck_failures_total{to=...} for the same upstream.

5. Check memory correlation. Pull go_memstats_heap_inuse_bytes and process_resident_memory_bytes over the same window. Goroutines climbing with flat heap leans toward transient blocking. Goroutines climbing with monotonically rising post-GC heap minimums leans hard toward a leak, and tells you the runway to the OOM kill.

6. Profile if it is a leak. If the count never drains, capture a goroutine profile from the Go pprof endpoint (served by the pprof plugin at http://localhost:6053/debug/pprof/goroutine?debug=1 by default). A leak shows thousands of goroutines parked in the same stack frame, usually a channel receive or network read inside one plugin. That stack tells you which plugin and whether it matches a known fix.

7. Rule out the DoQ cases. If you serve DNS-over-QUIC and run a version before 1.12.2 (CVE-2025-47950) or before 1.14.3 (CVE-2026-32934), unbounded stream goroutines are a documented remote-trigger condition. Treat this as a security issue, not just a capacity issue.

Metrics and signals to monitor

Fixes

Slow upstream causing blocked calls

The direct fix is restoring upstream health: remove or replace the slow upstream in the Corefile forward block. The per-upstream to label tells you which one. If the slowness is transient (network path, upstream overload), the backlog drains on its own once latency normalizes.

If you need guardrails against recurrence, the forward plugin’s max_concurrent option caps in-flight forwarded queries and rejects excess with REFUSED. That is deliberate backpressure: bounded failures instead of unbounded goroutine growth. Size it at roughly 3x your peak QPS multiplied by average upstream latency. The tradeoff is that excess queries fail fast instead of waiting, which is usually what you want from a shared resolver.

A restart will clear the accumulated goroutines, but treat it as a last resort: it also flushes the cache, which creates a cold-cache thundering herd on your upstreams at the worst possible moment. If the upstream has recovered, the backlog drains without a restart.

Confirmed goroutine leak

There is no config fix for a plugin leak. The path is: identify the leaking stack from a pprof profile, match it to a known issue, and upgrade. Known fixed cases include the transfer plugin AXFR leak (1.12.4), the secondary plugin reload leak (1.13.2), and the DoQ stream goroutine exhaustion fixes (1.12.2 and 1.14.3). If your stack does not match a known fix, capture the profile and report it upstream with the version.

As an interim measure while you schedule the upgrade, you can raise the container memory limit to extend the runway, but that only delays the OOM kill. A leak does not self-heal. If the leak rate is fast, scheduled rolling restarts, staggered to avoid cold-cache stampedes, are a defensible temporary mitigation.

Memory pressure compounding the problem

If heap is climbing with the goroutines, you are on the OOM clock regardless of cause. Estimate runway as (container limit minus current RSS) divided by the post-GC minimum growth rate. If the runway is shorter than your time to fix, raise the limit temporarily as a defensive move while you work the actual cause.

Prevention

• Alert on the shape, not just the value. Two alerts: goroutines >2x baseline sustained 10 minutes (catches both patterns early), and goroutines >5000 with rising RSS (catches the leak-before-OOM case). A single static threshold misses slow leaks under the line.
• Monitor per-upstream latency as a leading indicator. Blocked-call accumulation always starts as upstream latency. Alerting on one upstream drifting 3x slower than its peers catches the problem before goroutines pile up.
• Keep CoreDNS current. The documented goroutine leaks are all version-specific and fixed. Running an old version with the transfer, secondary, or quic plugins active is running known-leaky code.
• Set max_concurrent deliberately if your upstreams are unreliable. Bounded REFUSED responses are a better failure mode than unbounded goroutine accumulation when an upstream drags.
• Baseline your normal. Record the idle goroutine count, QPS, and heap for each deployment variant you run. “2x baseline” is only actionable if you know the baseline.
• Track restarts against goroutine history. If a pod was OOMKilled, check whether goroutines were climbing before the kill. That retro-check often reveals a slow leak that restarts were masking.

How Netdata helps

• Netdata charts go_goroutines at per-second resolution, which makes the distinguishing shapes visible: a spike that tracks a traffic burst and drains, versus a ratchet that never returns to baseline.
• Correlating goroutine count with go_memstats_heap_inuse_bytes and process_resident_memory_bytes on one dashboard separates “blocked but transient” from “leaking toward OOM” without switching tools.
• Per-upstream latency from coredns_proxy_request_duration_seconds{to=...} sits next to the goroutine curve, so you can see which upstream started dragging before the accumulation began.
• GC pause duration (go_gc_duration_seconds) alongside heap shows the memory-pressure feedback loop that turns a goroutine problem into a latency problem.
• Anomaly detection on go_goroutines flags deviations from the learned baseline, which catches slow leaks that sit under static thresholds for days.

Related guides

• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS cache collapse: the cold-cache thundering herd after a rollout
• CoreDNS cache evictions: the cache is too small for the working set
• CoreDNS cache hit ratio dropping: latency and upstream load climbing together
• CoreDNS not resolving external domains: the missing catch-all forward zone
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed
• CoreDNS GC pauses adding tail latency: go_gc_duration_seconds and heap pressure
• CoreDNS high request latency: reading P99 by zone to find the cause
• How CoreDNS actually works in production: the plugin chain mental model
• CoreDNS monitoring checklist: the signals every production resolver needs
• CoreDNS monitoring maturity model: from survival to expert
• CoreDNS NOERROR with zero answers: the resolution failure that reports success