$ guides / coredns / coredns-high-request-latency-p99 ▌

Operations Guides

CoreDNS high request latency: reading P99 by zone to find the cause

P99 on coredns_dns_request_duration_seconds is at 400ms when it used to sit at 15ms. Do not start with the aggregate number: aggregate latency in CoreDNS blends at least two very different workloads. Cache hits should complete in single-digit milliseconds. Forwarded and Kubernetes-backed queries depend on systems outside CoreDNS itself.

The zone label on the duration histogram is the fork in the road. High latency in cluster.local points at the Kubernetes path. High latency in forwarded zones points at upstream resolvers. High latency everywhere, at every percentile, points at the CoreDNS process itself: CPU saturation or GC pressure. Until you split the histogram by zone, you are guessing.

This article walks through reading coredns_dns_request_duration_seconds per zone, interpreting the P50/P99 shape, and following each branch to its root cause.

What this means

coredns_dns_request_duration_seconds is a Prometheus histogram with labels server, zone, and view. There is no type label on this metric in any released version (the metrics README lists type for it, but the implemented label set never included it). It measures server-side processing time only: from the moment a request enters the plugin chain to the moment the response leaves it.

That scope matters. The metric does not include kernel socket buffer wait, conntrack drops, or network transit between client and CoreDNS. If clients report 5-second timeouts while CoreDNS reports a clean 25ms P99, the problem is in front of CoreDNS, not inside it. Keep that in mind before you tear into the plugin chain.

Within what the metric does measure, the percentile shape carries the diagnosis:

Elevated P99 with a normal P50 means most queries are fine but a tail is slow. Classic causes: slow upstreams, intermittent GC pauses, or one degraded zone mixed into an aggregate.
Elevated P50 means the typical query is slow. That is systemic: CPU saturation, CFS throttling, or an aggregate dominated by cache misses.

Reference points: cache hits should be sub-10ms. Sustained P99 above 100ms in a mostly-cached environment warrants a ticket. Sustained P99 above 500ms is strong degradation. If P50 itself is climbing, treat it as urgent.

Common causes

Cause	What it looks like	First thing to check
Slow upstream resolver	High P99 in forwarded zones only; `cluster.local` unaffected	`coredns_proxy_request_duration_seconds` by `to` label
Kubernetes API slowness	High latency isolated to `cluster.local` zone	`coredns_kubernetes_rest_client_request_duration_seconds` and API errors
Cache collapse or low hit ratio	Latency up across forwarded zones; cache hit ratio dropped	`coredns_cache_hits_total` / `coredns_cache_requests_total`
Go GC pauses	P99 spikes with normal P50, correlating with GC activity	`go_gc_duration_seconds`
CPU saturation / CFS throttling	All percentiles shifted right, all zones	Pod CPU usage vs limit, throttling counters
One slow upstream dragging the pool	High P99 in forwarded zones, one `to` label much worse	Per-upstream latency breakdown

Quick checks

All of these are read-only and safe to run during an incident.

# Pull the raw duration histogram and look at the zone breakdown
curl -s http://localhost:9153/metrics | grep 'coredns_dns_request_duration_seconds'

# Per-upstream latency (which upstream is slow)
curl -s http://localhost:9153/metrics | grep 'coredns_proxy_request_duration_seconds'

# Cache effectiveness
curl -s http://localhost:9153/metrics | grep -E 'coredns_cache_(hits|requests)_total'

# GC pauses and goroutine accumulation
curl -s http://localhost:9153/metrics | grep -E '^(go_gc_duration_seconds|go_goroutines)'

# Kubernetes API latency and error codes
curl -s http://localhost:9153/metrics | grep 'coredns_kubernetes_rest_client'

# Functional check: time a real lookup per zone
dig @<coredns-ip> kubernetes.default.svc.cluster.local   # cluster.local path
dig @<coredns-ip> example.com                            # forwarded path

Both dig calls print Query time in the answer footer, so compare the two values directly.

In Prometheus, the per-zone P99 query is the workhorse:

histogram_quantile(0.99,
  sum by (zone, le) (
    rate(coredns_dns_request_duration_seconds_bucket[5m])
  )
)

Run the same query at 0.50 and compare the two lines per zone. The gap between them tells you whether you are chasing a tail problem or a systemic one.

One instrumentation gotcha before you trust the breakdown: the zone label is populated per server block by the metrics plugin. If your Corefile has multiple server blocks but declares the prometheus plugin in only one of them, requests from the other zones are not reported under their own zone. Verify the plugin is declared in each server block before drawing conclusions from a suspiciously empty zone breakdown.

How to diagnose it

flowchart TD
  A[P99 alert on request duration] --> B{Split histogram by zone}
  B -->|P50 also elevated, all zones| C[Systemic: CPU saturation or CFS throttling]
  B -->|High latency in cluster.local only| D[Kubernetes API path]
  B -->|High latency in forwarded zones only| E[Upstream path]
  D --> D1[Check API latency and error codes]
  E --> E1[Check per-upstream latency by 'to' label]
  E --> E2[Check cache hit ratio]
  C --> C1[Check pod CPU vs limit and go_gc_duration_seconds]

Step 1: Split by zone. Query the histogram grouped by zone at P50 and P99. You are looking for which of three shapes you have: one hot zone, all zones hot, or all zones with a fat tail but a healthy median.

Step 2a, if the hot zone is cluster.local: The Kubernetes plugin answers from an in-memory snapshot built from API watches, so query-time latency here usually means the process is starved or the plugin is degraded, not that every query calls the API. Check coredns_kubernetes_rest_client_requests_total by code for 5xx or connection errors, and coredns_kubernetes_rest_client_request_duration_seconds for API slowness. Confirm with a direct lookup: dig @<coredns-ip> kubernetes.default.svc.cluster.local and compare the query time against an external name.

Step 2b, if the hot zones are forwarded: Break down the forward plugin’s per-upstream latency by the to label: coredns_proxy_request_duration_seconds{proxy_name="forward"} on CoreDNS 1.11.0+, coredns_forward_request_duration_seconds on 1.5 through 1.10. If one upstream is 10x slower than the others, you have found it. Also check cache hit ratio: a drop means more queries are being forwarded, which raises average latency even if every upstream is healthy.

Step 2c, if everything is slow including P50: Look at the process. Check pod CPU against its limit. In Kubernetes, hitting the CPU limit triggers CFS throttling, which shows up as erratic latency spikes rather than a smooth increase. Check go_gc_duration_seconds for pauses above 10ms, which land directly on P99. Check go_goroutines: a count climbing well above baseline without a matching QPS increase means queries are piling up behind something slow, usually an upstream.

Step 3: Rule out what the metric cannot see. If CoreDNS-side latency is clean but clients still see slow DNS, the problem is in front of the process: kernel UDP receive buffer drops (netstat -su, look for RcvbufErrors), conntrack exhaustion (/proc/sys/net/netfilter/nf_conntrack_count vs nf_conntrack_max), or plain network transit. Do not keep tuning CoreDNS when the packets never arrive.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`coredns_dns_request_duration_seconds` by `zone`	The primary latency signal; zone label splits Kubernetes vs upstream causes	P99 > 100ms sustained; P50 rising
`coredns_proxy_request_duration_seconds` by `to`	Isolates which upstream is slow	P99 > 250ms on any single upstream
Cache hit ratio (`hits` / `requests`)	Low hit ratio forces expensive forwarded queries	Drop > 20% from rolling baseline
`go_gc_duration_seconds`	GC pauses add directly to tail latency	Pauses > 10ms
`go_goroutines`	Blocked queries accumulate as goroutines	Sustained > 2x baseline
`coredns_kubernetes_rest_client_requests_total` by `code`	API health for the `cluster.local` zone	Any sustained 5xx or 403
`coredns_forward_max_concurrent_rejects_total`	Forward plugin backpressure under slowness	Any nonzero sustained rate

Fixes

Slow upstream

If one upstream is slow, remove it from the forward line in the Corefile temporarily and let the healthy upstreams carry the load. If all upstreams are slow, the problem is likely the network path or a shared bottleneck such as a cloud resolver rate limit, not the resolvers themselves. Verify with dig @<upstream-ip> example.com +time=2 +tries=1 from the CoreDNS pod before blaming the upstream operator.

Tradeoff: dropping an upstream reduces redundancy. Treat it as a temporary measure while you fix or replace it.

Kubernetes API path

If API error codes or latency explain the cluster.local latency, the fix is on the API server side: etcd health, API server load, network policy between CoreDNS and the API server, or RBAC (a 403 means the ServiceAccount lost permissions). CoreDNS serves stale data during an API disconnect, so resolution of existing names keeps working while new services silently stop appearing.

Cache-driven latency

If the latency appeared right after a rollout or reload, you are looking at a cold cache, not a real regression. It self-corrects in minutes. If cache evictions are climbing at steady state, the cache is undersized for the working set; raise the cache size in the Corefile. Do not restart pods to “fix” latency: a restart flushes the cache and makes it worse.

Process-level saturation

If CPU is the constraint, raise the CPU limit or reduce plugin-chain cost. Throttling at the limit produces exactly the erratic tail latency you are chasing. If GC pauses correlate with the P99 spikes, reduce heap pressure: shrink an oversized cache, and investigate goroutine accumulation from blocked upstream calls. Restarting is a last-resort emergency measure when goroutines are growing unbound, and it only buys time.

Prevention

Dashboard the split, not the aggregate. Per-zone P50 and P99 should be a standing dashboard, not something you build during an incident. The aggregate hides the very signal you need.
Alert on shape, not just value. P99 > 100ms sustained is a ticket; P50 rising is the more urgent condition. Baseline-relative alerts (2x rolling P99) catch slow degradations that fixed thresholds miss.
Watch upstreams independently. Per-upstream latency and health check failures let you retire a degrading upstream before it drags the pool.
Protect the cache. Stagger rollouts (maxUnavailable=1 or a PodDisruptionBudget) so you never cold-start every replica at once.
Size CPU for DNS. Keep CoreDNS below roughly 60% of its CPU limit so CFS throttling never enters the latency picture.
Correlate with deploys. Latency that starts at a rollout boundary is usually cold cache or a new query pattern, not a CoreDNS bug.

How Netdata helps

Netdata charts CoreDNS request duration broken down by zone, so the P50/P99 split per zone is visible without writing PromQL mid-incident.
Per-upstream latency is charted by the to label, making the single-slow-upstream case obvious at a glance.
Go runtime signals (GC pause duration, goroutine count, heap) sit on the same dashboard as DNS latency, so the GC-vs-upstream distinction is a visual correlation instead of two separate tools.
Cache hit ratio and eviction rate alongside latency let you confirm or rule out a cold-cache cause in seconds after a rollout.
Because Netdata also monitors the node, UDP buffer errors and conntrack utilization are available when CoreDNS-side metrics look clean but clients still hurt.