CoreDNS memory limit and the re-list spike: sizing for the restart peak

A CoreDNS pod that runs fine for weeks gets OOMKilled thirty seconds after a restart. Kubernetes starts it again. It re-syncs, allocates heavily, crosses the limit, and gets killed again. Restart count climbs into the hundreds, and cluster DNS capacity drops by half or more while the crash loop runs. kubectl describe pod shows Last State: Terminated, Reason: OOMKilled, and nothing in steady-state monitoring predicted it.

The mechanism: the memory limit was sized against steady-state RSS, but the kubernetes plugin’s startup re-list is the actual peak memory event in CoreDNS’s lifecycle. In large clusters that peak runs 2-3x steady state. A limit set 10-20% above normal RSS is a limit sized to guarantee a crash loop on the next restart.

This article covers what happens during the restart, how to measure your actual re-list peak, how to size the limit and the request, and why adding replicas does nothing for this failure mode.

What happens during a restart

On every cold start, the kubernetes plugin has an empty in-memory snapshot. Before it can answer cluster.local queries, it performs a full list of all Services, Endpoints or EndpointSlices, and (if pods verified is configured) Pods from the API server, then populates its record set. This is the watch-based model working as designed: CoreDNS does not query the API per DNS query, it maintains a snapshot, and building that snapshot from zero requires materializing the entire object set in memory.

Two things make this expensive:

• The full list response is held in memory at once. The list is deserialized and held while the plugin builds the new store. The old snapshot and the new deserialization overlap, which roughly doubles the footprint before the store is swapped.
• Deserialization overhead lands on top of live data. JSON decoding, string allocations, and intermediate objects all allocate. In coredns issue #2511, a cluster with about 5,000 endpoints showed steady-state usage around 300 MB but allocations approaching 1 GB during initialization, traced to endpoint unmarshalling and strings.Join work.

A few properties of this peak matter operationally:

• It scales with cluster size, not with query load. QPS is irrelevant. What matters is the number and size of Services, Endpoints, and Pods. Large Endpoints objects (a Service with 150+ backends) make it worse.
• It happens on every cold start and every watch re-list. Not just pod restarts. If the watch to the API server drops past its timeout, the plugin does a full re-list. An API server restart can trigger simultaneous re-lists across every CoreDNS pod in the cluster. In kubernetes issue #139117, an apiserver restart in a scalability test restarted 235 of 313 CoreDNS pods, 233 of them OOMKilled (exit 137) at a 170 Mi limit.
• The pod is briefly alive between kills. It reaches Running, starts the sync, allocates, and dies. That is what makes the loop self-perpetuating: every restart recreates the exact conditions that caused the first kill.

flowchart TD
  A[Pod starts] --> B[kubernetes plugin begins full API list]
  B --> C[All Services, Endpoints, Pods deserialized in memory]
  C --> D[Memory climbs to 2-3x steady state]
  D --> E{Limit sized for peak?}
  E -->|No| F[OOMKilled - restart count increments]
  F --> A
  E -->|Yes| G[Snapshot built - memory settles to steady state]

Why steady-state sizing fails

The common mistake, called out in the CoreDNS operations playbook as one of the things most teams get wrong: set the memory limit 10-20% above observed RSS and move on. The observed RSS in a long-running pod is steady state. The re-list peak is invisible in that measurement because it only occurs at startup and during watch re-lists.

Issue coredns#3388 is the textbook case: a 170 Mi limit (the historical default in many manifests), roughly 150 Mi of observed pod memory in kubectl top, and 836 restarts per pod. The limit looked generous against normal operation and was nowhere near enough for startup.

There is a second trap: the official scaling formula from the CoreDNS deployment repo, MB required = (Pods + Services) / 1000 + 54, was derived from CoreDNS 1.2.x on Kubernetes 1.12. It does not account for the re-list peak, EndpointSlice overhead, or the autopath plugin’s memory cost. Treat it as a floor for steady state, not a limit recommendation.

The peak is per-pod and proportional to the whole cluster’s object set. Cluster growth is what turns a previously safe limit into a crash loop: the cluster adds Services and Endpoints for six months, steady-state RSS creeps up, the re-list peak creeps up 2-3x faster, and the next routine restart (a node upgrade, a config change, an eviction) tips the pod over.

Measuring your re-list peak

Do not guess from formulas. Measure the peak on your cluster. All commands below are read-only unless noted.

The CoreDNS image is minimal and typically has no shell or wget, so scrape the metrics endpoint from your workstation via kubectl port-forward rather than exec-ing into the container.

1. Get your steady-state baseline.

# Steady-state RSS (this is what the OOM killer sees)
kubectl port-forward -n kube-system deploy/coredns 9153:9153 &
PF_PID=$!
curl -s localhost:9153/metrics | grep '^process_resident_memory_bytes'
kill $PF_PID

2. Estimate the object set that drives the peak.

# Count the objects the kubernetes plugin must list
kubectl get svc --all-namespaces --no-headers | wc -l
kubectl get endpoints --all-namespaces --no-headers | wc -l
kubectl get endpointslice --all-namespaces --no-headers | wc -l

3. Capture the peak across a controlled restart.

Pick one pod, delete it, and scrape process_resident_memory_bytes from the replacement pod at 1-2 second intervals from the moment it starts until it reports ready. The maximum you observe is your re-list peak.

# Delete one replica, then watch memory climb on the replacement
kubectl delete pod -n kube-system <coredns-pod-name>
NEW_POD=$(kubectl get pods -n kube-system -l k8s-app=kube-dns \
  --sort-by=.metadata.creationTimestamp -o name | tail -1)

# Wait for the container to be Running (not Ready; ready comes after the sync)
kubectl wait -n kube-system $NEW_POD \
  --for=jsonpath='{.status.phase}'=Running --timeout=60s

kubectl port-forward -n kube-system $NEW_POD 9153:9153 &
PF_PID=$!
while true; do
  curl -s --max-time 1 localhost:9153/metrics | grep '^process_resident_memory_bytes'
  sleep 1
done
# Ctrl-C to stop, then: kill $PF_PID

Early curl calls fail silently until the metrics listener is up; that is fine, the climb is what you are after. If the pod gets OOMKilled during your test before you capture a max, that is also your answer: the peak exceeds the current limit. Raise the limit substantially (2x or more) and repeat until the pod survives, then read the peak.

This test removes one replica temporarily; run it only with a healthy second replica in place.

4. Check your history for evidence you already have a problem.

# Restart counts and last termination reason
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl describe pod -n kube-system <coredns-pod-name> | grep -A5 'Last State'

Any restart with Reason: OOMKilled within seconds to a minute of container start is the re-list pattern, not a slow leak. A slow leak kills a pod hours into its life; the re-list kill happens during initialization.

Sizing the limit and the request

Once you have a measured peak, sizing is arithmetic:

• Limit: at least 1.5x the measured re-list peak. The peak grows with cluster size between today and the next restart, and you will not get paged for “peak got slightly bigger” until the pod crash loops. If you cannot measure directly, the playbook guidance of at least 50% headroom over steady state is a minimum for small clusters; in large clusters where the re-list runs 2-3x steady state, size for 3x steady state plus margin. For a pod at 70 MB steady state, that means 210 MB or more, not 85 MB.
• Request: steady-state RSS plus margin, not the peak. The request drives scheduling and QoS. Setting the request at the peak wastes node capacity for a once-per-restart event; setting it at steady state keeps the pod schedulable and keeps it out of BestEffort. Request and limit must be different numbers here. A pod with request = limit at steady-state sizing will crash loop exactly like an undersized limit.
• Set GOMEMLIMIT below the limit. On Go 1.19 and later, GOMEMLIMIT is a soft memory limit that makes the GC work harder as usage approaches it. Set it to roughly 80-90% of the container limit. During the re-list this buys you GC pressure before the kernel’s OOM killer gets involved, which is often the difference between surviving the peak and not.
• Recheck after cluster growth events. The peak is a function of live object count. A workload onboarding that adds thousands of Endpoints, or switching the kubernetes plugin to pods verified mode (which adds a full Pod watch), changes the peak materially. The plugin documentation itself warns that pods verified “requires substantially more memory.”

Two mitigations reduce the peak itself rather than accommodating it:

• noendpoints disables the Endpoints watch, removing the largest object type from the list. Only viable if nothing depends on headless Service or endpoint DNS records.
• EndpointSlices instead of Endpoints (default in current versions) produces more fine-grained watch data and avoids the giant Endpoints-object deserialization cost documented in issue #2511. If you are on an older CoreDNS pinned to the Endpoints API, upgrading changes your peak.

Whether either eliminates the spike entirely versus merely shrinking it is cluster-dependent; measure before and after if you change these.

Why more replicas do not help

Horizontal scaling fixes throughput problems. This is not a throughput problem.

Every CoreDNS replica builds a complete, independent snapshot of the cluster. The re-list peak is per-pod and identical on every replica. Adding a third or fourth replica adds three or four pods that will each individually crash loop on their next restart. It also makes the worst case worse: when the API server restarts and every replica re-lists simultaneously, more replicas means more concurrent full-list responses hammering the apiserver and more pods racing toward the same OOM kill.

The correct response is vertical: raise the per-pod memory limit to cover the peak, set GOMEMLIMIT, and reduce the object footprint the plugin must list. Replicas are for query capacity and availability, not for memory headroom.

Signals to monitor

How Netdata helps

• Netdata charts process_resident_memory_bytes per pod at per-second resolution, which is what you need to actually see the re-list spike; at 15-60 second scrape intervals the peak can fall between samples and the crash loop looks unmotivated.
• Correlating RSS with pod restart events on one timeline separates the restart-peak OOM (vertical climb at t=0, killed within a minute) from a heap leak (slow climb over hours), which have completely different fixes.
• Tracking go_memstats_heap_inuse_bytes and go_gc_duration_seconds alongside RSS shows whether GOMEMLIMIT is doing its job during startup or whether GC pressure is arriving too late.
• Alerting on RSS as a percentage of the container limit, rather than an absolute value, keeps the threshold correct as you resize.
• Long retention on per-pod memory lets you watch the re-list peak grow quarter over quarter as the cluster adds Services, so you resize the limit before a routine node drain turns into an outage.

Related guides

• CoreDNS memory climbing: heap growth, post-GC minima, and leak detection
• CoreDNS GC pauses adding tail latency: go_gc_duration_seconds and heap pressure
• CoreDNS cache collapse: the cold-cache thundering herd after a rollout
• CoreDNS goroutine count climbing: blocked upstream calls and leaks
• How CoreDNS actually works in production: the plugin chain mental model