$ guides / coredns / coredns-goroutine-leak ▌

Operations Guides

CoreDNS goroutine count climbing: blocked upstream calls and leaks

Your go_goroutines graph for CoreDNS is trending up. Maybe it spiked during an incident and never came back down. Maybe it has been climbing slowly for days. Either way, the question is the same: are queries piling up behind a slow upstream, or is something leaking goroutines that will never exit?

The distinction matters because the fixes are completely different. A blocked-upstream spike resolves when the upstream recovers. A leak grows until the OOM killer ends the debate for you.

What this means

CoreDNS handles each in-flight DNS query on its own goroutine, plus a small set of background goroutines for cache maintenance, Kubernetes API watches, health checks, and metrics. Go’s runtime spawns and reaps these cheaply, so the goroutine count is a live proxy for concurrent work.

A healthy instance sits at a baseline of roughly 20-50 goroutines. Under load the count tracks approximately with QPS multiplied by average query duration: fast queries mean goroutines are created and destroyed so quickly the count stays low. Two failure shapes break this pattern:

Blocked upstream calls. A query that needs forwarding holds its goroutine until the upstream responds or the forward timeout fires. If an upstream gets slow but not dead, in-flight queries accumulate. Goroutines climb, latency climbs with them, and each goroutine holds memory. This pattern is self-limiting: when the upstream recovers, the backlog drains and the count returns to baseline.
A goroutine leak. A goroutine blocked on something that will never complete: an unclosed connection, a deadlocked channel, a plugin bug. The count ratchets upward regardless of load and never returns to baseline. At roughly 4KB minimum stack per goroutine plus per-goroutine allocations, enough leaked goroutines push the process toward its memory limit and an OOM kill.

The operational dividing line: a sustained count above 2x your stable baseline for more than 10 minutes warrants investigation. Above 5000 goroutines combined with rising memory is a strong leak signal. A startup spike that settles within about 30 seconds is normal Go runtime behavior, not a leak.

Common causes

Cause	What it looks like	First thing to check
Slow upstream (blocked forward calls)	Goroutines climb alongside P99 latency; count drops when upstream recovers	Per-upstream latency: `coredns_proxy_request_duration_seconds{to=...}`
Upstream connection churn	Connection cache misses rising; FDs climbing with goroutines	`coredns_proxy_conn_cache_misses_total` and `process_open_fds`
Traffic burst / retry storm	Goroutines scale proportionally with QPS; no leak signature	Query rate vs goroutine count ratio
Plugin goroutine leak	Count ratchets up independent of QPS; never returns to baseline	pprof goroutine profile; CoreDNS version vs known leak fixes
DoQ stream exhaustion (CVE-2025-47950, CVE-2026-32934)	Explosive goroutine growth on versions before 1.12.2 / 1.14.3 with DoQ enabled	CoreDNS version and whether the `quic` Corefile block is in use
Memory pressure feedback loop	Goroutines, heap, and GC pauses all rising together	`go_memstats_heap_inuse_bytes` and `go_gc_duration_seconds`

The version-specific leaks are worth naming because the fix is an upgrade, not a config change: the transfer plugin leaked a goroutine on AXFR errors before 1.12.4, and the secondary plugin leaked one on reload before 1.13.2. The DoQ server spawned an unbounded goroutine per QUIC stream before 1.12.2, and a regression (CVE-2026-32934) kept spawning unbounded waiter goroutines until 1.14.3. If you are on an affected version and the leak signature matches, upgrading is the fix.

Quick checks

These are all read-only and safe to run against a production instance.

# Current goroutine count
curl -s http://localhost:9153/metrics | grep '^go_goroutines'

# Per-upstream forward latency (which upstream is slow?)
curl -s http://localhost:9153/metrics | grep 'coredns_proxy_request_duration_seconds'

# Per-upstream health check failures
curl -s http://localhost:9153/metrics | grep 'coredns_proxy_healthcheck_failures_total'

# Forward concurrency backpressure (if max_concurrent is set)
curl -s http://localhost:9153/metrics | grep 'coredns_forward_max_concurrent_rejects_total'

# Heap and GC state
curl -s http://localhost:9153/metrics | grep -E '^(go_memstats_heap_inuse_bytes|go_gc_duration_seconds|process_resident_memory_bytes)'

# Open file descriptors (connection leak signature)
curl -s http://localhost:9153/metrics | grep '^process_open_fds'

# Query rate for proportionality check
curl -s http://localhost:9153/metrics | grep '^coredns_dns_requests_total'

Note on metric names: newer CoreDNS versions renamed the proxy plugin to forward, so on recent releases the per-upstream metrics appear as coredns_forward_request_duration_seconds, coredns_forward_healthcheck_failures_total, and coredns_forward_conn_cache_misses_total. If a grep returns nothing, try the other prefix before concluding the signal is absent.

Also check the CoreDNS version against the leak fixes listed above, and look at the pod restart count in Kubernetes: repeated OOMKilled restarts with a climbing goroutine history before each kill is the leak archetype reaching its conclusion.

How to diagnose it

The core diagnostic is shape analysis: does the curve track load, or does it ratchet?

flowchart TD
  A[go_goroutines climbing] --> B{Tracks QPS and returns to baseline?}
  B -- Yes --> C[Blocked upstream calls]
  B -- No, ratchets up --> D[Goroutine leak]
  C --> E[Find slow upstream via per-upstream latency]
  E --> F[Remove or replace slow upstream]
  D --> G[Capture goroutine profile with pprof]
  G --> H{Stack trace points to a plugin?}
  H -- Yes --> I[Check version vs known leak fixes]
  H -- No --> J[Check DoQ enabled + version vs DoQ CVEs]
  I --> K[Upgrade CoreDNS]
  J --> K

Establish the baseline. Look at go_goroutines over the last 24-72 hours, not just the incident window. You need to know what “normal” is for this instance before judging the spike. Baselines of 20-50 are typical; yours may differ with heavy watches or DoQ enabled.
Test proportionality. Overlay the goroutine count against query rate (coredns_dns_requests_total) and P99 latency (coredns_dns_request_duration_seconds). If goroutines grew 10x while QPS grew 2x, queries are blocking, not just arriving. If goroutines grow with no QPS change at all, suspect a leak immediately.
Check the recovery behavior. This is the single most diagnostic observation: after the load or latency spike ends, does the count return to baseline? Blocked-call accumulation drains within minutes of the upstream recovering. A leak does not drain. Watch the 10-30 minutes after the event.
Identify the slow upstream. If it is blocked calls, break down coredns_proxy_request_duration_seconds by the to label. One upstream at 500ms+ while others are at 20ms tells you exactly where the backlog is forming. Cross-check coredns_proxy_healthcheck_failures_total{to=...} for the same upstream.
Check memory correlation. Pull go_memstats_heap_inuse_bytes and process_resident_memory_bytes over the same window. Goroutines climbing with flat heap leans toward transient blocking. Goroutines climbing with monotonically rising post-GC heap minimums leans hard toward a leak, and tells you the runway to the OOM kill.
Profile if it is a leak. If the count never drains, capture a goroutine profile from the Go pprof endpoint (if enabled in your Corefile via the pprof plugin). A leak shows thousands of goroutines parked in the same stack frame, usually a channel receive or network read inside one plugin. That stack tells you which plugin and whether it matches a known fix.
Rule out the DoQ cases. If you serve DNS-over-QUIC and run a version before 1.12.2 (CVE-2025-47950) or before 1.14.3 (CVE-2026-32934), unbounded stream goroutines are a documented remote-trigger condition. Treat this as a security issue, not just a capacity issue.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`go_goroutines`	The symptom itself; proxy for concurrent in-flight work	>2x baseline sustained 10 min; >5000 with rising memory
`coredns_proxy_request_duration_seconds{to=...}`	Per-upstream latency; finds the slow upstream causing blocking	One upstream 3-5x slower than the others
`coredns_dns_request_duration_seconds`	Overall P99; confirms user impact of blocked queries	P99 climbing in step with goroutines
`go_memstats_heap_inuse_bytes`	Leak confirmation; goroutine stacks and allocations consume heap	Post-GC minimum rising and never reclaiming
`process_resident_memory_bytes`	What the OOM killer actually measures	Trending toward container memory limit
`coredns_forward_max_concurrent_rejects_total`	Backpressure signal if `max_concurrent` is configured	Any sustained nonzero rate
`process_open_fds`	Connection leak signature alongside goroutine leak	Climbing without returning to baseline
`go_gc_duration_seconds`	Heap pressure feedback into tail latency	Pauses increasing as goroutines climb

Fixes

Slow upstream causing blocked calls

The direct fix is restoring upstream health: remove or replace the slow upstream in the Corefile forward block. The per-upstream to label tells you which one. If the slowness is transient (network path, upstream overload), the backlog drains on its own once latency normalizes.

If you need guardrails against recurrence, the forward plugin’s max_concurrent option caps in-flight forwarded queries and rejects excess with REFUSED. That is deliberate backpressure: bounded failures instead of unbounded goroutine growth. Size it at roughly 3x your peak QPS multiplied by average upstream latency. The tradeoff is that excess queries fail fast instead of waiting, which is usually what you want from a shared resolver.

A restart will clear the accumulated goroutines, but treat it as a last resort: it also flushes the cache, which creates a cold-cache thundering herd on your upstreams at the worst possible moment. If the upstream has recovered, the backlog drains without a restart.

Confirmed goroutine leak

There is no config fix for a plugin leak. The path is: identify the leaking stack from a pprof profile, match it to a known issue, and upgrade. Known fixed cases include the transfer plugin AXFR leak (1.12.4), the secondary plugin reload leak (1.13.2), and the DoQ stream goroutine exhaustion fixes (1.12.2 and 1.14.3). If your stack does not match a known fix, capture the profile and report it upstream with the version.

As an interim measure while you schedule the upgrade, you can raise the container memory limit to extend the runway, but that only delays the OOM kill. A leak does not self-heal. If the leak rate is fast, scheduled rolling restarts, staggered to avoid cold-cache stampedes, are a defensible temporary mitigation.

Memory pressure compounding the problem

If heap is climbing with the goroutines, you are on the OOM clock regardless of cause. Estimate runway as (container limit minus current RSS) divided by the post-GC minimum growth rate. If the runway is shorter than your time to fix, raise the limit temporarily as a defensive move while you work the actual cause.

Prevention

Alert on the shape, not just the value. Two alerts: goroutines >2x baseline sustained 10 minutes (catches both patterns early), and goroutines >5000 with rising RSS (catches the leak-before-OOM case). A single static threshold misses slow leaks under the line.
Monitor per-upstream latency as a leading indicator. Blocked-call accumulation always starts as upstream latency. Alerting on one upstream drifting 3x slower than its peers catches the problem before goroutines pile up.
Keep CoreDNS current. The documented goroutine leaks are all version-specific and fixed. Running an old version with the transfer, secondary, or quic plugins active is running known-leaky code.
Set max_concurrent deliberately if your upstreams are unreliable. Bounded REFUSED responses are a better failure mode than unbounded goroutine accumulation when an upstream drags.
Baseline your normal. Record the idle goroutine count, QPS, and heap for each deployment variant you run. “2x baseline” is only actionable if you know the baseline.
Track restarts against goroutine history. If a pod was OOMKilled, check whether goroutines were climbing before the kill. That retro-check often reveals a slow leak that restarts were masking.

How Netdata helps

Netdata charts go_goroutines at per-second resolution, which makes the distinguishing shapes visible: a spike that tracks a traffic burst and drains, versus a ratchet that never returns to baseline.
Correlating goroutine count with go_memstats_heap_inuse_bytes and process_resident_memory_bytes on one dashboard separates “blocked but transient” from “leaking toward OOM” without switching tools.
Per-upstream latency from coredns_proxy_request_duration_seconds{to=...} sits next to the goroutine curve, so you can see which upstream started dragging before the accumulation began.
GC pause duration (go_gc_duration_seconds) alongside heap shows the memory-pressure feedback loop that turns a goroutine problem into a latency problem.
Anomaly detection on go_goroutines flags deviations from the learned baseline, which catches slow leaks that sit under static thresholds for days.