$ guides / coredns / coredns-dns-programming-duration ▌

Operations Guides

CoreDNS DNS programming latency: how long a Service takes to become resolvable

You create a Service, the API server accepts it, kubectl get svc shows it, and yet for some period of time nothing in the cluster can resolve its name. That gap, from “the object exists in the API” to “CoreDNS answers for it”, is DNS programming latency, and CoreDNS exposes it as a first-class metric: coredns_kubernetes_dns_programming_duration_seconds.

This is the Kubernetes DNS Programming SLI. It exists because service discovery delay is a real failure mode: a rollout completes, endpoints are ready, but clients still get NXDOMAIN or stale answers for seconds or tens of seconds. From the application’s perspective the new pods are unreachable by name. From CoreDNS’s request metrics everything looks green.

This article covers what the metric actually measures, the boundaries it does not cross (API processing, etcd, kubelet time), its known coverage limitations by service kind, and how to read it alongside API latency and error signals so you can tell “CoreDNS is slow to ingest changes” apart from “the control plane is slow to deliver them.”

What it is and why it matters

coredns_kubernetes_dns_programming_duration_seconds is a Prometheus histogram exported by the kubernetes plugin when the prometheus plugin is enabled. Each observation is the time between CoreDNS’s watch delivering a Service/Endpoint change event and that change being reflected in CoreDNS’s in-memory DNS record set. It carries a service_kind label with three documented values: cluster_ip, headless_with_selector, and headless_without_selector.

Collect it the same way as any CoreDNS metric (metrics are served on port 9153 by default; use kubectl port-forward or the pod IP if you are not on the CoreDNS host):

# Raw histogram buckets
curl -s http://localhost:9153/metrics | grep 'coredns_kubernetes_dns_programming_duration_seconds'

Why it matters: readiness probes, load balancers, and controllers all converge quickly; if DNS lags behind them, you get a window where the platform says “ready” and clients say “no such host.” These windows are exactly what this metric makes visible. A P99 above 30 seconds means some service changes take half a minute to become resolvable, which is long enough to break deploy pipelines, HPA-driven scale-ups, and failover flows.

How it works

The kubernetes plugin does not query the API server per DNS request. It maintains persistent watches on the API and builds an in-memory record set from watch events. The programming duration metric times one hop of that pipeline: from the watch event arriving at CoreDNS to the internal record set being updated.

flowchart LR
  A[kubectl apply / controller] -->|write| B[API server]
  B -->|persist| C[etcd]
  B -->|watch event| D[CoreDNS kubernetes plugin]
  D -->|update in-memory records| E[DNS answers]
  F[kubelet / endpoint controllers] -->|status updates| B
  subgraph measured["Measured by dns_programming_duration"]
    D
  end
  subgraph excluded["Excluded from the metric"]
    B
    C
    F
  end

Two consequences fall out of this design:

The metric is a watch-lag measure, not an end-to-end measure. It excludes API server processing time, etcd propagation, and kubelet or controller time. If the API server is slow to accept or fan out the change, that delay happens before the clock starts for this histogram. You will not see it here.
A low value does not guarantee clients see fresh data. If the cache plugin is in the chain, positive and negative responses are served from cache. CoreDNS may update its internal record set instantly (low programming duration) while clients still receive a cached answer, including a cached negative answer from before the Service existed. The metric measures the kubernetes plugin’s ingestion, not what the cache serves afterward.

Coverage limitations by service kind

This is the part that changes how much trust you can place in the metric.

headless_with_selector: the case where the metric has historically worked reliably. This is the acknowledged baseline for trustworthy data.
cluster_ip: for years the histogram silently recorded nothing for ClusterIP services, even though the label value was documented. A fix landed in March 2026 via CoreDNS PR #7951, shipping in the v1.14.3 timeframe, so that cluster_ip observations are now recorded. If you run an older CoreDNS and see zero observations for service_kind="cluster_ip", that is the known gap, not a healthy cluster.
headless_without_selector: still not covered. The plugin README lists this under known bugs. Do not expect observations for manually-managed endpoint services.

Practical reading: on CoreDNS versions before the ClusterIP fix, treat this metric as a headless-service-only signal. On current versions, check that you actually see observations for both cluster_ip and headless_with_selector before building alerts on it. A flat zero on a label you expect to be populated is an instrumentation gap, not good news.

Where it shows up in production

High programming duration is rarely a CoreDNS compute problem. The usual drivers:

API server pressure delaying watch delivery. If the watch stream is backed up or the API server is slow to fan out events, events arrive at CoreDNS late. This shows up either as genuinely delayed delivery or as CoreDNS grinding through a burst of events after a reconnect and re-list.
Large cluster churn. Mass service creation, big rollouts, or HPA events that touch many endpoints at once produce event bursts. Processing the burst takes time, and tail observations spike.
Post-reconnect catch-up. After a watch disconnection, CoreDNS re-lists and resynchronizes. During catch-up, programming duration for queued changes can look terrible even though steady-state behavior is fine.

The user-visible symptom is always the same shape: “the Service exists but the name does not resolve yet” or “the new endpoints exist but the headless service still returns the old pod IPs.” If you also see API request errors from the kubernetes plugin, suspect you are drifting toward the stale-data failure mode covered in CoreDNS Kubernetes API disconnect, where existing names keep resolving from stale state while new ones go invisible.

Reading it: thresholds and interpretation

Treat the histogram as a latency SLO for service discovery:

P50 under 1s, P99 under 5s: healthy. Changes propagate about as fast as anyone expects.
P99 above 30s: concerning. Service discovery is significantly delayed; some clients will see failures during deploys and scale events.
P99 above 60s: stale data is causing real problems. Assume deploys and failovers are partially broken.

Always read it per service_kind, and always with a count check. A histogram with a handful of observations per hour will produce wild quantile noise; look at the _count series to make sure you have enough volume to trust the quantiles.

Two false-alarm patterns to discount:

Startup and reconnect windows. During initial sync and post-reconnect catch-up, elevated values are expected. CoreDNS delays serving at startup until watches sync, bounded by a startup timeout of 5 seconds by default. Quantiles during that window do not reflect steady state.
Burst events. A single mass-change event (a 500-endpoint scale-up) legitimately takes longer to process. Sustained elevation matters; a single spike aligned with a known deploy usually does not.

Correlating with API latency and errors

Because the metric starts its clock at CoreDNS, you need adjacent signals to locate the delay in the full pipeline. The two that matter:

coredns_kubernetes_rest_client_request_duration_seconds: latency of the plugin’s calls to the API server. If programming duration is high AND API request latency is high, the control plane is the bottleneck. If programming duration is high but API latency is normal, suspect CoreDNS-side event processing or event bursts.
coredns_kubernetes_rest_client_requests_total by code: sustained 5xx or connection errors mean the watch path itself is unhealthy. In that case programming duration data becomes sparse and untrustworthy, and your real problem is the disconnect, not the latency.

When the metrics are ambiguous, a functional end-to-end check cuts through all of it. Run it from somewhere that can reach the CoreDNS Service ClusterIP (a pod, or a node with cluster DNS routing):

# Time how long a new Service takes to become resolvable.
# Creates and deletes a throwaway Service; harmless, but bounded by timeout
# because the loop would otherwise run forever if resolution never happens.
kubectl create service clusterip test-dns-delay --tcp=80:80
time timeout 30 sh -c 'until dig @<coredns-ip> test-dns-delay.default.svc.cluster.local +short +time=1 +tries=1 | grep -q .; do sleep 0.2; done'
kubectl delete service test-dns-delay

This measures the true end-to-end programming time including API and etcd time, which the histogram deliberately excludes. If the functional test is fast but the histogram is slow, look at CoreDNS event processing. If the functional test is slow but the histogram is fast, the delay is upstream of CoreDNS.

Signals to watch in production

Signal	Why it matters	Warning sign
`coredns_kubernetes_dns_programming_duration_seconds` P99	Direct measure of DNS programming latency	P99 > 30s sustained
`_count` series per `service_kind`	Tells you whether quantiles have enough observations, and whether a service kind is silently uninstrumented	Zero observations for a kind you expect (version gap or unsupported kind)
`coredns_kubernetes_rest_client_request_duration_seconds`	Separates control-plane delay from CoreDNS ingestion delay	P99 > 1s sustained
`coredns_kubernetes_rest_client_requests_total` by `code`	Watch/API health; errors make programming data unreliable	Sustained 5xx or any 403
DNS request latency for `cluster.local`	Client-visible impact of stale or delayed records	Rising P99 in the Kubernetes zone alongside programming duration

How Netdata helps

Netdata charts coredns_kubernetes_dns_programming_duration_seconds per second, so programming latency spikes line up visually with the deploy or scale event that caused them.
Splitting the histogram by service_kind makes the coverage gaps obvious: you can see at a glance that cluster_ip has zero observations on an older CoreDNS instead of misreading it as instant propagation.
Correlating programming duration with coredns_kubernetes_rest_client_request_duration_seconds and API request error codes in one dashboard is what separates “API server is slow” from “CoreDNS is slow to ingest” without manual metric spelunking.
ML anomaly detection on the programming duration quantiles flags sustained regime changes (for example after a cluster grows past a size threshold) rather than single deploy bursts, which are usually noise.
Pairing it with cluster.local zone request latency shows the client-visible consequence of programming delay, which is the number your incident review actually cares about.