CoreDNS pod alive but not ready: the /ready endpoint and API sync

The pod is Running. The process is up, the logs show CoreDNS started, maybe it is even answering some queries. But kubectl get pods -n kube-system shows 0/1 READY, the pod is excluded from the kube-dns Service endpoints, and your cluster is running on one fewer DNS replica than you think. If this is your only replica, or the other one is on the same node, you are one hiccup away from a cluster-wide DNS outage.

This almost always comes down to one thing: the readiness probe hits /ready on port 8181, and the ready plugin will not return 200 until the kubernetes plugin has finished syncing its watches against the Kubernetes API server. A pod that cannot reach the API, cannot authenticate, or cannot complete the initial list-and-watch stays not-ready indefinitely. The process is alive. The plugin is not.

A few seconds of not-ready at startup is normal. Not-ready for more than about 60 seconds after the process started is a fault, and the fault is almost always between CoreDNS and the API server, not inside the DNS serving path.

What this means

CoreDNS exposes two different HTTP probes, and they answer different questions:

• /health on port 8080: process liveness. Returns 200 as long as the CoreDNS process is running. It does not test plugin state, and it does not test DNS resolution.
• /ready on port 8181: plugin-aware readiness. Returns 200 only once every plugin that implements readiness has signaled ready. For the kubernetes plugin, “ready” means the initial API list-and-watch sync has completed.

This split exists for a reason. On startup, the kubernetes plugin must build its in-memory record set by listing Services, Endpoints (EndpointSlices in current versions), and optionally Pods, then establishing watches. Until that sync completes, CoreDNS cannot authoritatively answer cluster.local queries. CoreDNS delays serving Kubernetes-zone answers up to startup_timeout (5 seconds by default) while syncing; if sync does not finish in that window, it starts anyway but returns SERVFAIL for records that have not synced.

The Kubernetes readiness probe gates whether the pod appears in the kube-dns Service endpoints. Not-ready means the pod correctly receives no traffic. That is the system working as designed. The bug is whatever is preventing sync.

One more behavior worth internalizing before you debug: during graceful shutdown with lameduck configured on the health plugin, /health starts returning 503 while the process keeps answering queries for the lameduck duration. That is intentional: the pod should drain from endpoints while it finishes in-flight work. So a pod briefly going not-ready during a rolling update is expected behavior, not a fault.

flowchart TD
  A[kubelet readiness probe GET :8181/ready] --> B{ready plugin}
  B --> C{all plugins ready?}
  C -->|yes| D[200 OK - pod enters kube-dns endpoints]
  C -->|no| E[503 - pod stays out of endpoints]
  F[kubernetes plugin HasSynced] --> C
  G[K8s API list + watch] -->|syncs records| F
  G -.->|unreachable or 403 or slow| H[sync never completes - stuck not ready]

Common causes

Quick checks

All read-only. Note that the stock CoreDNS container image is scratch-based and has no shell, wget, or curl, so kubectl exec will not work against it. Use the API server proxy (kubectl get --raw) or kubectl port-forward instead.

# 1. Confirm which pods are not ready and for how long
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

# 2. Query the ready endpoint directly, via the API server pod proxy
kubectl get --raw "/api/v1/namespaces/kube-system/pods/<coredns-pod>:8181/proxy/ready"
# Body "OK" = ready. A 503 body lists plugins that are not ready.

# 3. Compare with the health endpoint (process liveness)
kubectl get --raw "/api/v1/namespaces/kube-system/pods/<coredns-pod>:8080/proxy/health"
# If this returns OK but /ready fails, the process is fine and plugin sync is the problem.

# 4. Look for watch/list errors and readiness retries in the logs
kubectl logs -n kube-system <coredns-pod> --tail=200 | grep -iE "watch|list|error|retry|ready"

# 5. Check API client errors by HTTP status code
kubectl get --raw "/api/v1/namespaces/kube-system/pods/<coredns-pod>:9153/proxy/metrics" | grep 'coredns_kubernetes_rest_client_requests_total'
# code="403" = RBAC. code="5xx" = API server trouble.

# 6. Test API reachability from a throwaway pod in the same namespace
kubectl run apitest --rm -i --restart=Never -n kube-system --image=curlimages/curl -- curl -sk https://kubernetes.default.svc/healthz

# 7. Check what the readiness probe is actually pointing at
kubectl describe pod -n kube-system <coredns-pod> | grep -A4 Readiness

Check 7 matters more than it looks. If the readiness probe points at /health instead of /ready, you have the opposite problem: pods marked ready before they can answer cluster.local queries, which is how you get SERVFAIL during rollouts. AWS EKS switched its default to /ready after exactly this class of incident.

How to diagnose it

1. Establish the timeline. How long has the pod been not-ready? Under startup_timeout (5s default) is normal. Under about 60 seconds in a large cluster may be slow sync. Beyond that, treat it as a fault. Also check whether the pod was ever ready: a pod that was ready and flipped is a different case from a pod that never became ready.

2. Split liveness from readiness. If /health (8080) returns OK but /ready (8181) does not, the process and the DNS listener are fine. You are debugging the kubernetes plugin’s API sync, full stop. If neither responds, you have a process or listener problem instead, and this article’s causes do not apply.

3. Read the logs for the sync failure. A pod stuck on API sync logs repeated readiness retries and watch/list errors (for example, failures to list Endpoints, or watch stream errors). The log content tells you which cause you are in: connection refused and timeouts point at connectivity; forbidden points at RBAC; TLS errors point at certificates or clock skew.

4. Corroborate with the API client metric. coredns_kubernetes_rest_client_requests_total breaks requests down by code, method, and host. code="403" confirms RBAC. Sustained 5xx or missing responses confirm API-side or network-side failure. If the counter is barely moving at all, the plugin may not even be attempting requests, which points at a stuck reflector.

5. Test connectivity and auth directly. From a test pod, hit the API healthz endpoint (check 6 above). If that fails, test DNS resolution of kubernetes.default.svc from another pod and check NetworkPolicies in kube-system: a policy that blocks CoreDNS pods from egress to the API server on 443 is a classic cause after a security tightening.

6. Check RBAC. The CoreDNS ServiceAccount needs list and watch on services, endpoints, namespaces, pods, and endpointslices.discovery.k8s.io. A ClusterRole that looks almost right but is missing one resource (commonly EndpointSlices after a version upgrade) produces exactly this symptom.

7. If the pod was ready and flipped after a reload, check for a stuck watch. There is a known failure pattern where a transient network error during the watch stream leaves the reflector unable to recover, and the pod sits not-ready forever. The only reliable recovery in that state is deleting the pod. This is disruptive to DNS capacity, so confirm the other replica is healthy and ready first.

Metrics and signals to monitor

Note what is absent: there is no binary “watch is broken” metric. Watch health has to be inferred from the API client error codes, programming duration, and logs.

Fixes

API connectivity or NetworkPolicy

Restore the path from the CoreDNS pods to the API server. If a NetworkPolicy was recently added or tightened in kube-system, that is your prime suspect; allow egress from CoreDNS pods to the API server. Verify with check 6 after the change. Tradeoff: none, this is purely corrective.

RBAC

Fix the ClusterRole to grant list and watch on all resources the plugin watches, including EndpointSlices. After fixing, delete the not-ready pod so it re-syncs cleanly. Deleting a CoreDNS pod reduces DNS capacity briefly, so do it one pod at a time and confirm the replacement becomes ready before touching another.

Slow sync in a large cluster

If the pod consistently becomes ready after 60-120 seconds and the cluster has thousands of Services and Endpoints, the initial list is just slow. Raise startup_timeout in the kubernetes plugin block so the window matches reality, and watch API request latency for signs the API server itself is overloaded. Tradeoff: a longer startup_timeout delays traffic to a legitimately slow pod; the readiness gate is what protects you, so size the probe timeouts accordingly rather than loosening them blindly.

TLS or clock skew

If logs or API-side errors point at certificates, check node time synchronization first. Skewed node clocks invalidate certificate validity windows and produce exactly this symptom. Fix the clock, renew expired certs, then restart the pod.

Stuck watch after reload

If the reflector is wedged, delete the pod. A fresh pod does a full re-list and re-establishes the watch. If this recurs on reloads, check your CoreDNS version; the stuck-reflector pattern was reported against older releases, and you should be on something current.

What not to do

Do not “fix” readiness by pointing the probe at /health. The pod will show ready, receive traffic, and return SERVFAIL for cluster.local until sync completes. You have converted a visible, correct signal into an invisible partial outage. Likewise, neither /ready nor /health tests actual DNS resolution; if you need that assurance, it has to come from a synthetic query check, not from probe configuration.

Also note that once the kubernetes plugin has synced, a later API disconnect does not necessarily flip the pod back to not-ready: the pod keeps serving from its in-memory record set even as the data goes stale. Whether and how the ready plugin re-checks plugins after initial readiness is version- and configuration-dependent. The stale-data failure mode after an API disconnect is a separate problem; see the related guide below.

Prevention

• Probe on /ready, not /health. Confirm the readiness probe in your CoreDNS Deployment targets port 8181 path /ready. Older Corefiles and some distributions historically omitted the ready plugin entirely; if it is missing from the Corefile, the endpoint does not exist and the probe fails open or closed depending on configuration.
• Size startup_timeout for your cluster. In clusters with thousands of Services, the default 5 seconds is not enough headroom. Measure actual time-to-ready and set it above the observed P99 sync time.
• Alert on not-ready duration. “Pod alive but not ready for more than 60 seconds” is a clean, low-noise alert that catches every cause in this article.
• Watch the API client metrics. Alert on any 403 and on sustained 5xx in coredns_kubernetes_rest_client_requests_total. RBAC regressions after upgrades are common and otherwise silent.
• Test RBAC in upgrade pipelines. EndpointSlice permission gaps appear after Kubernetes version changes. A post-upgrade check that a fresh CoreDNS pod becomes ready within N seconds catches this before the old pods cycle.
• Keep lameduck behavior in mind during rollouts. Not-ready during shutdown is correct draining behavior. If rolling updates cause DNS timeouts anyway, the lag is usually kube-proxy and iptables propagation after endpoint removal, not CoreDNS.

How Netdata helps

• Readiness duration tracking turns “pod stuck not ready” from something you notice in kubectl into an alert the moment it crosses your threshold.
• API client error codes from coredns_kubernetes_rest_client_requests_total are scraped per pod, so you can see at a glance whether a not-ready pod is hitting 403s, 5xx, or not talking to the API at all.
• Per-replica correlation matters here: one pod not-ready while its peer serves fine is invisible in aggregated metrics. Netdata keeps per-instance series so a single degraded replica stands out.
• SERVFAIL by zone lets you confirm whether an unready pod that received traffic anyway (probe misconfiguration) is actually hurting clients on cluster.local.
• Timeline correlation between pod events, Corefile reloads, and readiness transitions helps you catch the stuck-reflector-after-reload pattern instead of chasing it as a networking issue.

Related guides

• CoreDNS Kubernetes API disconnect: stale records and new services going invisible
• How CoreDNS actually works in production: the plugin chain mental model
• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS cache collapse: the cold-cache thundering herd after a rollout
• CoreDNS cache evictions: the cache is too small for the working set
• CoreDNS cache hit ratio dropping: latency and upstream load climbing together
• CoreDNS CPU throttling: CFS limits making a green dashboard lie about latency
• CoreDNS not resolving external domains: the missing catch-all forward zone
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed
• CoreDNS GC pauses adding tail latency: go_gc_duration_seconds and heap pressure
• CoreDNS goroutine count climbing: blocked upstream calls and leaks
• CoreDNS high request latency: reading P99 by zone to find the cause