CoreDNS memory climbing: heap growth, post-GC minima, and leak detection

CoreDNS memory has been creeping up for hours. The Grafana panel shows a sawtooth that never quite comes back down, and you need to answer one question before the pod gets OOMKilled: is this Go being Go, or is something actually leaking?

The trap is that go_memstats_heap_inuse_bytes is spiky by design. The heap grows, the garbage collector reclaims, the heap grows again. If you react to instantaneous peaks you will chase ghosts all night. The signal that separates normal GC behavior from a leak is the trend of the post-GC minima: the floor each sawtooth returns to. If that floor keeps rising over an hour or more and never returns to baseline, you have a genuine leak heading for the container memory limit.

The second trap is confusing heap with RSS. Heap in-use is what the Go runtime is actively managing. RSS (process_resident_memory_bytes) is what the kernel and the OOM killer see. They are different numbers, they answer different questions, and treating them as interchangeable leads to bad alert and capacity decisions in both directions.

What this means

CoreDNS is a Go process, so it inherits Go’s memory model. The runtime allocates heap for query handling, cache entries, the Kubernetes plugin’s in-memory snapshot of Services and Endpoints, goroutine stacks, and per-request buffers. The collector reclaims dead objects on its own schedule, and freed memory is returned to the OS lazily. The result is a sawtooth on any heap graph, with peaks that can sit far above the live data.

A leak here means memory that is still referenced and therefore cannot be reclaimed. The three leak shapes to check are goroutine-stack accumulation (goroutines blocked or leaked, each holding stack memory), unbounded cache growth (cache sized beyond what the container can hold, or query diversity exploding the working set), and connection leaks to upstreams (each leaked connection holds buffers and a file descriptor).

The failure mode is cliff-edge. There is no graceful degradation: RSS approaches the container limit, the OOM killer terminates the pod, and on restart the kubernetes plugin’s initial re-list can spike memory straight back into the limit, producing a crash loop. Your job is to catch the rising floor hours before the cliff.

Common causes

Version-specific leaks are real and fixed in specific releases. CoreDNS before 1.12.2 has a DNS-over-QUIC memory exhaustion issue (CVE-2025-47950, fixed in 1.12.2 with max_streams and worker_pool_size limits). Release 1.12.4 fixed a goroutine leak in the transfer plugin on AXFR error (PR #7516) and a span leak in the gRPC plugin (PR #7487). Release 1.13.2 fixed a secondary plugin goroutine leak on reload (PR #7694). If your version predates these and the leak shape matches, upgrading is the fix.

Quick checks

All of these are read-only. The metrics endpoint is on port 9153 by default.

# Heap in use (spiky: judge the floor, not the peaks)
curl -s http://localhost:9153/metrics | grep '^go_memstats_heap_inuse_bytes'

# RSS: the number the OOM killer cares about
curl -s http://localhost:9153/metrics | grep '^process_resident_memory_bytes'

# Goroutines: baseline is typically 20-50 at idle
curl -s http://localhost:9153/metrics | grep '^go_goroutines'

# GC pauses: increasing pauses track heap pressure
curl -s http://localhost:9153/metrics | grep '^go_gc_duration_seconds'

# Cache utilization: entries pinned at max means eviction pressure
curl -s http://localhost:9153/metrics | grep -E '^coredns_cache_(entries|evictions_total)'

# File descriptors: rising without fall suggests a connection leak
curl -s http://localhost:9153/metrics | grep -E '^process_(open|max)_fds'

Two supporting checks outside the metrics endpoint:

# Confirm OOM kills actually happened (Kubernetes)
kubectl describe pod -n kube-system <coredns-pod> | grep -A3 'Last State'

# Cluster growth check: is the snapshot growing?
kubectl get svc --all-namespaces | wc -l
kubectl get endpoints --all-namespaces | wc -l

How to diagnose it

Work through this in order. Each step either clears a suspect or hands you the next one.

1. Establish the floor trend. Graph go_memstats_heap_inuse_bytes over at least one hour, ideally longer. Read the minima of each GC trough, not the peaks. A flat floor with tall peaks is a healthy process under load. A floor that rises steadily and never returns to baseline is the leak signature. A useful alert-grade threshold: post-GC minimum growing more than 10 MB per minute sustained without returning to baseline.

2. Check OOM risk separately with RSS. Compare process_resident_memory_bytes against the container memory limit, not the heap number. Working limits: below 70% is the operating target, 80% is warning, 90% is critical. It is entirely possible to have a scary heap graph and comfortable RSS, or the reverse, because Go does not eagerly return freed memory to the OS. Heap answers “is something leaking”; RSS answers “how long until the OOM killer acts”.

3. Estimate runway. (container memory limit - current RSS) / growth rate per minute, using the post-GC minimum growth rate rather than peak growth. This tells you whether you have hours to diagnose properly or minutes to add headroom.

4. Correlate the floor with the three leak shapes. Pull go_goroutines, coredns_cache_entries, and process_open_fds onto the same time window as the heap floor. Whichever metric rises in lockstep with the floor names the leak family: goroutines for stack accumulation, cache entries for cache growth, FDs for connection leaks. If none of them track the floor, suspect the kubernetes plugin snapshot (compare against Service/Endpoint counts) or a plugin bug.

5. Capture heap profiles to pinpoint the source. Metrics tell you the leak family; pprof tells you the exact allocation site. Capture two heap profiles separated in time (for example, 30 minutes apart during the rising phase) and diff them. The diff shows which allocation paths grew between the two snapshots:

# Diff two heap profiles to isolate the growth
go tool pprof -base heap_1.pb.gz heap_2.pb.gz

Inside pprof, top and the inuse_space sample type show live heap by call stack. One caveat: the heap profile reads the low watermark right after a GC cycle, so with the default GC target the live process can be holding roughly double what the profile reports between cycles. Use the profile to identify the source, not to reconcile exactly against RSS. Also note that the pprof endpoint is not exposed by default; enabling it requires adding the pprof plugin to the server block in the Corefile (it listens on localhost:6053 by default, path /debug/pprof/) and a reload, which resets runtime stats. Capture your trend data from metrics before doing it.

6. Rule out the normal lookalikes. Before concluding “leak”, check the boring explanations: a recent load spike whose RSS has not decayed yet (Go returns memory lazily, so elevated RSS after a burst is normal if the floor is flat), cold-cache warmup after a restart, or goroutine count spiking at startup and settling within about 30 seconds.

flowchart TD
  A[Heap floor rising over 1h+] --> B{RSS vs container limit}
  B -->|">90%: critical"| C[Add memory headroom now]
  B -->|"Headroom OK"| D{Correlate floor with...}
  D -->|go_goroutines rising| E[Goroutine or stack leak]
  D -->|cache_entries at max| F[Cache oversized for limit]
  D -->|process_open_fds rising| G[Connection leak]
  D -->|None track| H[Snapshot growth or plugin bug]
  E --> I[pprof heap diff]
  G --> I
  H --> J[Check CoreDNS version vs fixed leaks]

Metrics and signals to monitor

Fixes

Goroutine or connection leak

If the leak family is goroutines or FDs and the process is degrading, a restart clears the accumulated state, but treat it as buying time, not a fix. Blocked goroutines on slow upstreams accumulate until memory runs out; the upstream slowness is the root cause and the leak will recur under the same conditions. If the leak shape matches a known version-specific bug (transfer plugin, gRPC plugin, secondary plugin, DoQ), the fix is upgrading to the release that carries the fix.

Cache oversized for the limit

Reduce the cache capacity in the Corefile so the steady-state working set plus GC headroom fits inside the container limit with margin. The tradeoff is direct: smaller cache means more evictions, lower hit ratio, higher upstream load and latency. Size from the memory limit downward, not from the hit-ratio wish list upward. The default capacity is 9984 entries per cache, and entries are counted by record, not by byte, so record-size distribution affects real memory per entry.

Kubernetes plugin snapshot growth

If the floor tracks cluster growth, the fix is raising the memory limit to match the cluster you actually have, with at least 50% headroom over steady-state RSS. The headroom is not optional: it covers GC overhead (Go can hold roughly double the live heap before collecting under the default target) and the re-list spike on restart, which is the peak memory event. A limit set 10-20% above steady state is what turns a routine restart into an OOM crash loop.

Emergency headroom

When runway is measured in minutes, temporarily raise the container memory limit while you diagnose. This is a stopgap: it converts an imminent OOM kill into time to find the source.

Prevention

• Track the floor, not the peaks. Alert on the post-GC minimum growth rate and on RSS relative to the limit. Never alert on instantaneous heap values; that is a false-positive machine.
• Size memory for the restart spike. Set the limit with at least 50% headroom over steady-state RSS so the kubernetes plugin re-list on restart cannot OOM the pod into a crash loop.
• Keep CoreDNS current. Several real leaks (transfer, gRPC, secondary, DoQ) are fixed in specific releases. Pin to a version that carries those fixes.
• Recheck sizing after cluster growth events. New Services and Endpoints grow the snapshot permanently. A limit that was right six months ago may be one rollout away from the cliff.
• Correlate replicas independently. Two CoreDNS replicas behind the Service can diverge; one leaking pod is invisible in averaged metrics.

How Netdata helps

• Netdata charts go_memstats_heap_inuse_bytes at high resolution alongside process_resident_memory_bytes, so the sawtooth peaks and the rising floor are visibly distinct on the same dashboard, and RSS can be read directly against the container limit.
• Correlating go_goroutines, coredns_cache_entries, and process_open_fds against the heap floor on one screen turns the “which leak family” step from a guess into a visual match.
• go_gc_duration_seconds next to request latency shows when heap pressure starts taxing tail latency before the OOM kill arrives.
• Pod restart events with OOMKilled reason overlaid on the memory trend confirm the cliff and anchor the post-incident timeline.

Related guides

• CoreDNS GC pauses adding tail latency: go_gc_duration_seconds and heap pressure
• CoreDNS cache evictions: the cache is too small for the working set
• CoreDNS cache hit ratio dropping: latency and upstream load climbing together
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed
• CoreDNS high request latency: reading P99 by zone to find the cause
• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS cache collapse: the cold-cache thundering herd after a rollout
• CoreDNS not resolving external domains: the missing catch-all forward zone
• CoreDNS NOERROR with zero answers: the resolution failure that reports success
• How CoreDNS actually works in production: the plugin chain mental model
• CoreDNS monitoring checklist: the signals every production resolver needs
• CoreDNS monitoring maturity model: from survival to expert