$ guides / coredns / coredns-gc-pause-tail-latency ▌

Operations Guides

CoreDNS GC pauses adding tail latency: go_gc_duration_seconds and heap pressure

Your CoreDNS dashboards look mostly fine: P50 latency is sub-millisecond, SERVFAIL is zero, cache hit ratio is healthy. But P99 is spiking to tens or hundreds of milliseconds at irregular intervals, and a small fraction of clients see DNS timeouts they cannot explain. The process never crashes. Upstream latency is clean. CPU looks acceptable on average.

This is the classic signature of Go garbage collection stop-the-world pauses landing on DNS query latency. CoreDNS is a Go process, and every in-flight query is a goroutine. When the GC stops the world, every one of those goroutines waits. DNS is supposed to be sub-millisecond for cache hits, so a pause that would be invisible in a batch service is user-visible here. The working threshold: GC pauses over 10ms are impactful for DNS and show up as P99 spikes.

The important framing: rising GC pause duration over hours is a symptom, not the disease. It signals growing heap pressure, usually cache growth or a leak, and it often precedes an OOM kill. The fix is almost always reducing the heap, not tuning GC parameters.

What this means

Go’s garbage collector does most of its work concurrently, but it still has stop-the-world phases. During those phases, no query goroutine makes progress. A query that arrived just before the pause simply sits. Because CoreDNS answers most queries in microseconds, a 10ms pause is orders of magnitude larger than normal service time, and it lands squarely in the tail.

The signal chain:

flowchart LR
  A[Heap grows: cache, leaks, watch snapshot] --> B[GC cycles get longer and more frequent]
  B --> C[go_gc_duration_seconds rises]
  C --> D[Stop-the-world pauses stall query goroutines]
  D --> E[coredns_dns_request_duration_seconds P99 spikes]
  A --> F[RSS approaches container limit]
  F --> G[OOMKilled - cliff-edge outage]

Two things to note. First, coredns_dns_request_duration_seconds measures CoreDNS processing time only; a GC pause also lets the kernel UDP receive buffer back up, so application-observed latency can be worse than the metric shows. Second, the same heap growth driving the pauses is driving you toward the memory limit, so this pattern is a leading indicator for the OOM kill pattern, not just a latency nuisance.

Common causes

Cause	What it looks like	First thing to check
Cache growth filling the heap	Heap climbs with `coredns_cache_entries`; pauses rise gradually over hours	`coredns_cache_entries` vs configured cache size
Memory leak (goroutine or plugin)	Post-GC heap minimum never returns to baseline even after load drops	`go_goroutines` trend; heap minima across GC cycles
Large cluster watch snapshot	High baseline heap from the kubernetes plugin’s in-memory state; pauses elevated even at low QPS	Service/Endpoint count vs memory limit
Upstream slowness piling up goroutines	Goroutine count and heap grow together during upstream latency events	`coredns_proxy_request_duration_seconds` per upstream
Container limit too tight for working set	RSS near the limit, GC working hard, pauses spiking before an OOM	`process_resident_memory_bytes` vs container limit

Quick checks

All read-only. Run against the metrics endpoint on port 9153.

# Current GC pause quantiles (summary metric; quantile="1" is the max in the window)
curl -s http://localhost:9153/metrics | grep 'go_gc_duration_seconds'

# Heap in use right now
curl -s http://localhost:9153/metrics | grep '^go_memstats_heap_inuse_bytes'

# RSS - what the OOM killer actually sees
curl -s http://localhost:9153/metrics | grep '^process_resident_memory_bytes'

# Goroutine count - leaks show as growth that never returns to baseline
curl -s http://localhost:9153/metrics | grep '^go_goroutines'

# Fraction of CPU spent on GC
curl -s http://localhost:9153/metrics | grep '^go_memstats_gc_cpu_fraction'

# Cache occupancy - the biggest heap consumer in most deployments
curl -s http://localhost:9153/metrics | grep '^coredns_cache_entries'

# Tail latency: this is a histogram, so grep returns buckets per zone.
# Compute actual P99 in Prometheus with histogram_quantile over the buckets.
curl -s http://localhost:9153/metrics | grep '^coredns_dns_request_duration_seconds_bucket'

Two interpretation rules that save time. First, go_memstats_heap_inuse_bytes is naturally spiky: it grows, then GC reclaims. Look at the trend of post-GC minima, not instantaneous peaks. Second, heap-in-use is not RSS. For OOM risk, process_resident_memory_bytes against the container limit is the number that matters.

How to diagnose it

Confirm the correlation. Pull P99 of coredns_dns_request_duration_seconds and go_gc_duration_seconds{quantile="1"} over the same window. If latency spikes align with pause spikes, you have your mechanism. If they do not align, look elsewhere: slow upstreams, CPU throttling, or kernel-level drops.
Check whether pauses are trending. A flat sub-millisecond pause profile is normal Go behavior. The actionable signal is pause duration increasing over hours, which indicates growing heap pressure.
Characterize the heap trend. Look at go_memstats_heap_inuse_bytes over 24 hours. If the post-GC floor keeps rising, you have growth that GC cannot reclaim: cache growth, a leak, or cluster growth expanding the kubernetes plugin snapshot. If the floor is flat but peaks are high, the container limit may simply be too close to the working set.
Identify the heap consumer. Check coredns_cache_entries against the configured cache size (default is 9984 items). Check go_goroutines for a leak pattern: a count that grows and never returns to baseline. In Kubernetes, check whether Service and Endpoint counts have grown recently.
Assess OOM runway. Compare process_resident_memory_bytes to the container memory limit. A practical headroom rule: keep RSS under 70% of the limit to account for GC overhead and burst allocations; 80% is warning, 90% is critical. Also account for the restart spike: after a restart the kubernetes plugin re-lists all objects, which is the peak memory event and can re-trigger an OOM immediately.
Rule out GC-side CPU starvation. If go_memstats_gc_cpu_fraction is elevated, the collector is burning CPU fighting allocation pressure. Above roughly 25%, GC is dominating the process and you are near a death spiral where GC slows queries, queries pile up, and allocation grows further.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`go_gc_duration_seconds`	Stop-the-world pauses land directly on DNS tail latency	Pauses > 10ms sustained; any rising trend over hours
`go_memstats_heap_inuse_bytes`	Tracks the heap pressure driving pause growth	Post-GC minimum rising > 10MB/min without returning to baseline
`process_resident_memory_bytes`	What the OOM killer sees	RSS > 80% of container limit
`coredns_dns_request_duration_seconds` P99	The user-visible symptom	P99 > 100ms sustained in a mostly-cached workload
`go_goroutines`	Leaks and upstream-blocking pileups grow the heap	> 2x baseline sustained, or never returning to baseline
`go_memstats_gc_cpu_fraction`	Shows GC CPU cost, a saturation leading indicator	Sustained elevation; > 25% is critical
`coredns_cache_entries`	Cache is usually the largest heap consumer	Entries pinned at configured maximum with rising evictions

Fixes

Group by cause. Restarting CoreDNS is not on this list as a first move: it clears the heap but also clears the cache, trading a latency problem for a thundering herd, and if the heap driver is still present the pauses return.

Reduce the cache footprint

If coredns_cache_entries is at its configured maximum and heap is climbing, the cache is either undersized for the working set (see cache evictions) or oversized for the container. For the GC-pause problem specifically, shrink what the heap must hold: reduce the configured cache size, or cap TTLs so entries age out faster. Tradeoff: a smaller cache lowers the hit ratio, which raises upstream load and average latency. You are trading mean latency for tail latency and OOM headroom. Verify the change by watching the post-GC heap floor flatten.

Fix the leak

If post-GC minima rise regardless of load, something holds references. Check go_goroutines first: a goroutine count that grows monotonically points to blocked or leaked goroutines, often from slow upstreams or a plugin bug. For a heap-level answer, take a heap profile to find the retaining path. There is no GC setting that fixes a leak; every hour of delay also shortens your OOM runway.

Raise the memory limit as a stopgap

If RSS is approaching the container limit and pauses are spiking, increasing the limit buys room immediately. This is a legitimate emergency measure, not a fix: more heap headroom means fewer, less desperate GC cycles, which reduces pause pressure. Set the limit with at least 50% headroom over steady-state RSS so a post-restart re-list spike cannot immediately OOM the pod again.

Set GOMEMLIMIT

For CoreDNS builds on Go 1.19 or later, the GOMEMLIMIT environment variable sets a soft memory limit that makes GC run earlier and more aggressively as memory approaches it, instead of letting RSS drift toward the hard cgroup limit. Setting it to roughly 80% of the container limit is a common pattern for keeping Go processes out of OOM territory. Tradeoff: more aggressive GC means more GC CPU, which itself adds latency. It protects against the cliff but does not reduce pause frequency if the heap driver is still growing. Verify the Go version your CoreDNS build uses before relying on this.

Reduce upstream-blocking goroutine growth

If heap spikes track upstream slowness, the heap growth is blocked query goroutines, not data. Address the upstream side: replace the slow upstream identified by coredns_proxy_request_duration_seconds{to=...}, and consider max_concurrent on the forward plugin as backpressure so accumulation fails fast (REFUSED) instead of inflating the heap. See forward max_concurrent rejects for that failure mode.

Prevention

Alert on the trend, not just the threshold. A static “> 10ms pause” alert catches the acute case. The higher-value alert is on rising post-GC heap minima and rising pause duration over hours, which fires days before the OOM.
Track RSS against the container limit continuously. Warning at 70-80%, critical at 90%. Pair it with growth-rate extrapolation so you know your runway in days.
Size the cache and the container together. Cache entries consume heap; pick a cache size that fits comfortably inside the limit with GC overhead and restart-spike headroom, not the other way around.
Watch goroutines as a leak detector. Any sustained divergence from baseline after load events deserves investigation before it becomes heap pressure.
Stagger restarts. If you do restart CoreDNS to clear acute heap pressure, restart one replica at a time so the cold-cache thundering herd does not become your next incident.

How Netdata helps

Netdata collects the standard Go runtime metrics from the CoreDNS Prometheus endpoint, including go_gc_duration_seconds, go_memstats_heap_inuse_bytes, go_goroutines, and process_resident_memory_bytes, at per-second granularity, so short pause spikes that minute-resolution scrapes average away stay visible.
Plotting GC pause duration against coredns_dns_request_duration_seconds P99 on one dashboard makes the correlation step of diagnosis immediate instead of an export-and-compare exercise.
Heap trend views make the post-GC minimum visible, which is the specific signal that separates a leak or cache growth from normal Go spikiness.
RSS tracked against the container memory limit, with anomaly detection on the growth rate, surfaces OOM runway erosion while there is still time to act.
Goroutine count alerting catches the leak pattern early, before it converts into heap pressure and pause growth.