CoreDNS /health vs /ready: the readiness-probe mistake that serves SERVFAIL

Your CoreDNS pods look healthy. Liveness passes, readiness passes, the pods are Running and Ready. Yet for the first seconds or minutes after every pod start, clients get SERVFAIL for cluster.local names. New deployments fail to resolve their dependencies, retries pile up, and then the problem vanishes on its own.

This is the classic symptom of pointing the Kubernetes readiness probe at /health instead of /ready. The two endpoints answer different questions, and confusing them makes your readiness gate meaningless.

What this means

CoreDNS exposes two HTTP probes:

• /health on port 8080 is a process-liveness check. It returns 200 OK as soon as the CoreDNS process is running. It does not test plugin initialization, and it does not test DNS resolution.
• /ready on port 8181 is a plugin-aware readiness check. It returns 200 OK only after every plugin that implements readiness has reported ready. For the kubernetes plugin, that means the initial watch sync against the API server has completed.

On startup, CoreDNS delays serving Kubernetes records for up to 5 seconds while it lists and watches Services, Endpoints, and Pods (configurable as the kubernetes plugin startup_timeout option since v1.12.3; hardcoded at 5 seconds before that). In large clusters, sync can take much longer. If sync has not completed, or fails within that window, CoreDNS starts serving anyway and answers SERVFAIL for any cluster.local record it has not yet synchronized. External names handled by the forward plugin may already resolve, so you get a confusing partial failure: the internet works, your cluster does not.

If your readiness probe hits /health, the pod flips to Ready the instant the process starts. Kubernetes adds it to the kube-dns Service endpoints, kube-proxy starts sending it real client traffic, and that traffic gets SERVFAIL until the sync finishes. AWS EKS switched its managed CoreDNS add-on to /ready after production incidents caused by exactly this misconfiguration.

flowchart TD
  A[CoreDNS process starts] --> B[/health returns 200 immediately]
  A --> C[kubernetes plugin syncing watches]
  B --> D{readiness probe target?}
  D -->|/health - wrong| E[Pod marked Ready]
  E --> F[Traffic routed to pod]
  F --> G[SERVFAIL for cluster.local until sync done]
  D -->|/ready - correct| H[Pod stays unready]
  C -->|sync complete| I[/ready returns 200]
  H --> I
  I --> J[Pod marked Ready, safe traffic]

Common causes

Quick checks

All of these are read-only. Note that the CoreDNS container image is scratch-based: it has no shell and no wget, so probe and metrics checks run from a throwaway pod against the CoreDNS pod IP.

# 1. Inspect the probes on the CoreDNS deployment
kubectl get deploy coredns -n kube-system -o jsonpath='{.spec.template.spec.containers[0].livenessProbe.httpGet}'
echo
kubectl get deploy coredns -n kube-system -o jsonpath='{.spec.template.spec.containers[0].readinessProbe.httpGet}'
echo
# Correct: liveness -> path /health port 8080, readiness -> path /ready port 8181

# 2. Confirm the ready plugin is in the Corefile
kubectl get cm coredns -n kube-system -o jsonpath='{.data.Corefile}'
# Look for a line containing "ready"

# 3. Hit both endpoints directly from a debug pod and compare
IP=$(kubectl get pod -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[0].status.podIP}')
kubectl run -it --rm probecheck --image=busybox:1.28 --restart=Never -- \
  wget -qO- http://$IP:8080/health
kubectl run -it --rm probecheck2 --image=busybox:1.28 --restart=Never -- \
  wget -qO- http://$IP:8181/ready

# 4. Check how long a fresh pod takes to become ready
kubectl get pods -n kube-system -l k8s-app=kube-dns -w

# 5. Look at SERVFAIL by zone around a pod start
kubectl run -it --rm metricscheck --image=busybox:1.28 --restart=Never -- \
  sh -c "wget -qO- http://$IP:9153/metrics | grep coredns_dns_responses_total | grep SERVFAIL"

# 6. Functional test: resolve a cluster name right after a pod starts
kubectl run -it --rm dnstest --image=busybox:1.28 --restart=Never -- \
  nslookup kubernetes.default.svc.cluster.local

The tell-tale log line when readiness is correctly gated but sync is stuck is [INFO] plugin/ready: Still waiting on: "kubernetes" (CoreDNS up to v1.12; v1.13 and later log plugin/ready: Plugins not ready: ["kubernetes"]). Seeing it on a long-running pod is a different problem (API server unreachable, RBAC, network policy), not a probe misconfiguration.

How to diagnose it

1. Read the probes. Confirm the readiness probe path and port. If it says /health on 8080, you have found the misconfiguration.
2. Read the Corefile. Confirm the ready plugin is present. Without it, nothing listens on 8181 and a /ready probe will fail permanently.
3. Correlate SERVFAIL with pod starts. Pull coredns_dns_responses_total{rcode="SERVFAIL"} broken out by zone. If zone="cluster.local." SERVFAILs spike for seconds to minutes after each pod start and then stop, that is the sync window leaking traffic. Compare with coredns_dns_responses_total{rcode="SERVFAIL", plugin="forward"} to separate cluster-zone sync problems from upstream forwarding failures (the forward plugin’s own coredns_forward_responses_total metric existed only up to v1.10.x; it was removed in v1.11.0).
4. Time the readiness transition. A pod that goes Ready 1-2 seconds after the container starts in a large cluster almost certainly is not gating on plugin sync. Real sync takes longer as Service and Endpoint counts grow.
5. Test resolution during the window. Start a fresh CoreDNS pod and immediately query a cluster.local name against its pod IP. SERVFAIL plus a passing readiness probe confirms traffic is arriving before sync.
6. Rule out the look-alike. If pods never become ready at all, check API server reachability from the pod and RBAC on the CoreDNS ServiceAccount. That is a sync failure, not a probe mistake.

Metrics and signals to monitor

Fixes

Point the readiness probe at /ready

The correct probe layout, matching the upstream CoreDNS deployment manifest:

livenessProbe:
  httpGet:
    path: /health
    port: 8080
    scheme: HTTP
readinessProbe:
  httpGet:
    path: /ready
    port: 8181
    scheme: HTTP

Keep /health for liveness. It answers “should Kubernetes restart this process”, and restarting is the right response to a wedged process. Use /ready for readiness. It answers “should this pod receive traffic”, and the answer is no until the kubernetes plugin has synced.

Add the ready plugin to the Corefile

The probe only works if the plugin is loaded. A minimal Corefile stanza needs ready alongside health:

.:53 {
    errors
    health {
        lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}

Change the ConfigMap and the Deployment together. The EKS add-on upgrade incident pattern was exactly this: the managed update switched the readiness probe to port 8181 but did not touch the user-owned ConfigMap, so pods hung at 0/1 Ready until operators added ready to the Corefile by hand. If you manage CoreDNS through an add-on, verify both halves after every upgrade.

Understand the lameduck asymmetry

With lameduck 5s on the health plugin, /health keeps returning 200 during graceful shutdown while /ready flips to non-OK. That is deliberate: /ready drops first so the pod leaves Service endpoints and stops receiving new queries, then the process finishes in-flight work during the lameduck window, then liveness finally fails. If you repurpose /health for readiness, you break this ordering and drop in-flight queries on every rollout.

Accept what neither endpoint proves

Both probes are internal-state checks. Neither sends a DNS query. A pod can pass /health and /ready while returning SERVFAIL for every query because all upstreams are down, the cache is serving poisoned negatives, or the API watch silently dropped and the data went stale. Probe status is a routing gate, not an availability signal. Real availability comes from coredns_dns_responses_total{rcode="SERVFAIL"} and functional resolution tests.

Prevention

• Codify the probe pair. Keep liveness /health:8080 and readiness /ready:8181 in version control and review any diff that touches them.
• Gate add-on upgrades. After any managed CoreDNS upgrade, diff both the Deployment probes and the Corefile ConfigMap before considering it done.
• Alert on the sync window leaking. A composite alert on cluster.local SERVFAIL, correlated with pod start events, catches regressions where someone reverts the probe.
• Alert on stuck readiness. Not-ready persisting beyond 60 seconds after start means sync problems (API reachability, RBAC, network policy), which will bite during the next scale-up or node failure.
• Test resolution, not just probes. A periodic synthetic query for a known cluster.local name per pod is the only check that proves the pod actually serves DNS.

How Netdata helps

• SERVFAIL by zone: Netdata charts coredns_dns_responses_total split by rcode and zone, so cluster.local SERVFAIL is visibly distinct from upstream forwarding failures.
• Pod lifecycle correlation: overlaying CoreDNS pod restarts and rollout events on SERVFAIL charts makes the “errors only right after pod start” pattern obvious in seconds.
• Per-second resolution: the sync window is often 5-30 seconds. Per-second collection catches transient SERVFAIL bursts that minute-scraped metrics smooth away.
• Readiness duration tracking: tracking how long pods spend unready after start establishes a baseline, so both instant-ready (wrong probe) and never-ready (broken sync) stand out as anomalies.

Related guides

• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS cache collapse: the cold-cache thundering herd after a rollout
• CoreDNS cache evictions: the cache is too small for the working set
• CoreDNS cache hit ratio dropping: latency and upstream load climbing together
• CoreDNS CPU throttling: CFS limits making a green dashboard lie about latency
• CoreDNS DNS programming latency: how long a Service takes to become resolvable
• CoreDNS not resolving external domains: the missing catch-all forward zone
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed
• CoreDNS GC pauses adding tail latency: go_gc_duration_seconds and heap pressure
• CoreDNS goroutine count climbing: blocked upstream calls and leaks
• CoreDNS high request latency: reading P99 by zone to find the cause
• How CoreDNS actually works in production: the plugin chain mental model