$ guides / coredns / coredns-oomkilled ▌

Operations Guides

CoreDNS OOMKilled: the memory cliff and the restart crash loop

You found the CoreDNS pods in kube-system with OOMKilled as the last terminated state. Maybe one pod restarted once and recovered. Maybe both replicas are flapping and cluster DNS is degraded or down. Either way, the pod status operators paste into a search bar is the same: Last State: Terminated, Reason: OOMKilled.

Memory exhaustion in CoreDNS is a cliff edge, not a slowdown. There is no degradation curve where latency rises and errors climb while you watch. RSS approaches the container memory limit, the kernel OOM killer terminates the process instantly, and every client served by that pod loses DNS until a replacement is ready. The warning phase exists only in your metrics, and only if you are watching the right ones.

The nastier variant is the restart crash loop. The pod is OOMKilled, Kubernetes restarts it, the kubernetes plugin does a full re-list of all Services, Endpoints, and Pods to rebuild its in-memory snapshot, and that startup re-list allocates 2-3x steady-state memory. If the limit was set just above steady state, the re-list peak blows straight through it and the pod is OOMKilled again before it ever becomes ready. This is not hypothetical: in the upstream 5,000-node scalability test (kubernetes/kubernetes#139117), CoreDNS at roughly 82MB steady state was consistently OOMKilled by the re-list after an apiserver restart at the default 170MB limit, and the fix was doubling the limit to 340MB. If your limit is tuned to steady state with 10-20% headroom, one restart can put you in the same loop.

What this means

The OOM killer does not watch Go heap. It watches RSS, which is process_resident_memory_bytes on the metrics endpoint. RSS is always larger than live heap (go_memstats_alloc_bytes) because the Go runtime holds freed memory instead of eagerly returning it to the OS. You can have a reasonable heap and still get killed because RSS, GC headroom, goroutine stacks, and runtime overhead pushed past the cgroup limit.

Four things consume CoreDNS memory in practice:

The kubernetes plugin’s in-memory snapshot of Services, Endpoints, and optionally Pods. This scales with cluster size and is the dominant consumer in large clusters.
The cache plugin’s positive and negative caches, proportional to configured size and average record size.
Goroutines. One per in-flight query plus background workers. Slow upstreams cause blocked goroutines to accumulate, and each one holds stack and per-request allocations.
Go runtime overhead: GC metadata, freed-but-unreturned memory, allocator behavior.

The crash loop mechanism:

flowchart TD
  A[Steady state RSS near limit] --> B[OOM kill: instant termination]
  B --> C[Pod restarts]
  C --> D[Kubernetes plugin full re-list]
  D --> E[Re-list peak: 2-3x steady state]
  E --> F{Peak under limit?}
  F -->|No| B
  F -->|Yes| G[Ready, serves DNS]
  G --> H[Growth: cluster size, cache, leaks]
  H --> A

The loop breaks only if the re-list peak fits under the limit, or if whatever grew memory past steady state is fixed.

Common causes

Cause	What it looks like	First thing to check
Limit sized for steady state, not the re-list peak	OOMKilled immediately after every restart; pod never reaches ready; restarts correlate with apiserver restarts or rollouts	Restart timing vs. OOMKilled events; cluster Service/Endpoint count
Cluster growth outpaced the limit	Slow RSS climb over weeks, then first OOMKill; more frequent as Services/Endpoints grow	`kubectl get svc --all-namespaces \| wc -l` trend vs. limit
Cache too large for the limit	Cache entries pinned at configured max, evictions high, RSS tracks cache size	`coredns_cache_entries` vs. Corefile `cache` size
Goroutine or connection leak	`go_goroutines` grows without returning to baseline; RSS climbs in step; not tied to QPS	`go_goroutines` trend over hours
Blocked goroutines from slow upstreams	Goroutine and memory growth correlates with elevated upstream latency	`coredns_proxy_request_duration_seconds` per upstream
Unbounded forward concurrency	Memory spikes under query floods; no `max_concurrent` set in Corefile	Corefile `forward` block

Quick checks

All read-only and safe to run during an incident.

# Pod status and restart counts
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Confirm OOMKilled and see exit details
kubectl describe pod -n kube-system <coredns-pod> | grep -A 10 "Last State"

# Recent OOM events
kubectl get events -n kube-system | grep -i oom

# Previous container logs (what happened right before the kill)
kubectl logs -n kube-system <coredns-pod> --previous --tail=100

The metrics below are on the CoreDNS metrics endpoint (:9153, when the prometheus plugin is enabled). Run them from wherever you normally scrape, for example via kubectl exec -n kube-system <coredns-pod> -- curl -s localhost:9153/metrics or a port-forward, not from a node shell.

# Current RSS: this is the number the OOM killer watches
curl -s http://localhost:9153/metrics | grep '^process_resident_memory_bytes'

# Goroutine count: leaks show unbounded growth away from baseline
curl -s http://localhost:9153/metrics | grep '^go_goroutines'

# Heap in use: for leak trend analysis, not OOM prediction
curl -s http://localhost:9153/metrics | grep '^go_memstats_heap_inuse_bytes'

# Cache occupancy vs configured maximum
curl -s http://localhost:9153/metrics | grep '^coredns_cache_entries'
curl -s http://localhost:9153/metrics | grep 'coredns_cache_evictions_total'

# Cluster size driving the kubernetes plugin snapshot
kubectl get svc --all-namespaces | wc -l
kubectl get endpoints --all-namespaces | wc -l

If you have metrics-server installed, kubectl top pod -n kube-system gives a quick current-usage reading, but it is a point sample. The trend before the kill matters more than the value after.

How to diagnose it

Confirm the kill reason. kubectl describe pod should show Reason: OOMKilled under Last State. If it shows something else (Error, Completed, loop detection), this is a different failure. CrashLoopBackOff with “Loop detected” in logs is a forwarding loop, not memory.
Distinguish the crash loop from a slow leak. Look at restart timing. If the pod is killed within seconds to a minute of starting, every time, you are hitting the re-list peak. If the pod runs for hours or days and memory climbs until the kill, you have growth: cluster expansion, cache pressure, or a leak.
Reconstruct the pre-kill memory trend. You need historical process_resident_memory_bytes for the killed pod. Post-GC minimums rising steadily means a leak or an undersized limit for the working set. A flat trend that spikes only at startup means the re-list peak is the whole problem.
Check goroutines. If go_goroutines climbed in step with RSS and never returned to baseline after load dropped, suspect a leak or blocked upstream calls. Correlate with per-upstream latency (coredns_proxy_request_duration_seconds{to=...}): slow-but-not-dead upstreams accumulate blocked goroutines, and each one holds memory.
Check the cache. Compare coredns_cache_entries against the size in your Corefile cache directive. Entries pinned at the maximum with a rising eviction rate means the working set exceeds the cache, but the cache itself is also consuming the heap it was allocated. The default cache is 9984 items per cache (success and denial are separate), roughly 30MB when fully populated. Against a 170Mi limit, that is a meaningful fraction.
Quantify the snapshot. Count Services and Endpoints. The official scaling guidance estimates required memory as (Pods + Services) / 1000 + 54 MB for a default deployment, and (Pods + Services) / 250 + 56 MB with autopath. If your measured steady state is well above the estimate, something else (cache, goroutines, leak) is contributing. If it matches and your limit is below the estimate plus re-list headroom, the limit is simply wrong.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`process_resident_memory_bytes` vs. container limit	This is what the OOM killer uses	RSS > 70% of limit: investigate. > 85%: critical
Post-GC heap minimum trend	Leak detection independent of Go’s spiky allocation	Minimum rising steadily, never reclaiming
`go_goroutines`	Blocked calls and leaks accumulate here first	Sustained > 2x baseline; growth without QPS growth
`go_gc_duration_seconds`	GC pauses lengthen as heap pressure grows	Pauses > 10ms; increasing trend
`coredns_cache_entries` vs. configured max	Cache at capacity is both a symptom and a memory consumer	Pinned at max with rising evictions
`coredns_proxy_request_duration_seconds{to=...}`	Slow upstreams cause the goroutine accumulation that becomes memory pressure	P99 > 250ms for any upstream
Pod restart count and OOMKilled events	The cliff edge itself	Any restart with OOMKilled reason

Fixes

Limit too small for the re-list peak

Increase the memory limit so the startup re-list peak fits with headroom. The re-list can reach 2-3x steady state in large clusters, so a limit 10-20% above steady state is a crash loop waiting for its first restart. This is the correct emergency fix during an active loop: kubectl edit deployment coredns -n kube-system and raise the limit. It restarts the pods, but they are already crash looping, so you are not making anything worse. Size the new limit from the scaling formula plus re-list headroom, not from whatever number was there before.

Cluster grew past the limit

Same fix, different root cause. Re-run the sizing estimate with current Pod and Service counts and set the limit accordingly. Then alert on the growth rate: linear extrapolation of steady-state RSS against the limit gives you runway in days, and you want to re-size before you arrive, not after.

Cache too large for the limit

Reduce the cache size in the Corefile, but understand the tradeoff: a smaller cache means more evictions, a lower hit ratio, more upstream load, and higher latency. The better resolution is usually to raise the limit to fit the cache your working set needs, using cache entries and eviction rate as the sizing evidence. Shrinking the cache to fit an undersized limit trades one failure mode (OOM) for another (upstream amplification).

Goroutine or connection leak

Confirm with the go_goroutines trend. If goroutines accumulate when upstreams are slow, the fix is at the upstream or network layer, and max_concurrent on the forward plugin gives you backpressure (rejected queries get REFUSED instead of unbounded goroutine growth; each in-flight query costs roughly 2KB plus stack). If the count grows independent of traffic, suspect a plugin bug: connection and goroutine leaks have been fixed across releases, so check your CoreDNS version against release notes and upgrade. For a persistent unidentified leak, enable pprof and capture a heap and goroutine profile to find the allocation source.

Go runtime behavior

If your CoreDNS is built with Go 1.19 or later, setting the GOMEMLIMIT environment variable to just below the container limit makes the GC work harder as RSS approaches the cap, at the cost of CPU. This is a mitigation, not a fix: in a GC death spiral you trade an OOM kill for CPU saturation. It buys time and smooths the approach to the cliff, but correct limit sizing is still the answer.

Prevention

Size for the peak, not the mean. Set the limit to cover the startup re-list peak: 2-3x steady-state RSS in large clusters, with a floor from the scaling formula. Keep steady-state RSS under 70% of the limit.
Alert on the approach, not the corpse. OOMKilled is a post-mortem signal. Alert on RSS > 70% of limit (investigate) and > 85% (critical), on post-GC heap minimum trending up, and on goroutine growth without QPS growth. These are the signals that exist while you can still act.
Track cluster growth against the limit. Service and Endpoint counts are leading indicators of snapshot size. When the cluster grows, re-size before the first OOMKill, not after.
Re-check sizing after topology changes. Rollouts that restart all CoreDNS pods simultaneously re-trigger the re-list peak on every pod at once and also cold-start every cache. Stagger rollouts (maxUnavailable=1, PodDisruptionBudget) so you never test the re-list peak cluster-wide.
Keep CoreDNS current. Memory-relevant fixes (connection leaks, socket caps, watch handling) ship in patch releases. Staying current removes known leak sources from the suspect list.
Mind the apiserver restart coupling. The re-list peak also fires when the apiserver restarts and watches reconnect. After control plane maintenance, watch CoreDNS RSS the same way you watch the apiserver.

How Netdata helps

RSS against the limit, per pod. Netdata charts process_resident_memory_bytes alongside container memory limits from cgroup data, so the approach to the cliff is visible as a percentage per CoreDNS replica, not as an aggregate that hides one dying pod.
Goroutine and heap correlation. Plotting go_goroutines, go_memstats_heap_inuse_bytes, and GC pause duration on the same timeline separates a leak (all climbing together) from a one-off spike (heap reclaims, goroutines return to baseline).
Cache pressure signals. coredns_cache_entries and coredns_cache_evictions_total per pod show whether the cache is the memory consumer and whether it is undersized for the working set.
Restart and OOMKill context. Pod restart counts and container state, correlated with the memory trend leading into each kill, tell you immediately whether you are looking at a slow leak or the re-list crash loop.
Upstream latency on the same dashboard. Per-upstream forward latency next to goroutine count confirms or rules out slow-upstream accumulation as the memory driver, without switching tools.