CoreDNS Kubernetes API disconnect: stale records and new services going invisible

The symptom that usually surfaces first is not a DNS error. It is a developer saying “I deployed the service twenty minutes ago and nothing can reach it,” while every existing workload looks fine. nslookup kubernetes.default works. External domains resolve. CoreDNS dashboards are green. But anything created or changed recently in the cluster does not exist as far as DNS is concerned.

This is the Kubernetes API disconnect pattern. The CoreDNS kubernetes plugin maintains its DNS records by watching the API server through informers, not by querying the API per DNS request. When that watch breaks and cannot reconnect, CoreDNS keeps answering from its last known in-memory snapshot. Existing services keep resolving, which is exactly why the failure is so dangerous: everything looks healthy while the answers drift further from reality.

There is no binary “watch broken” metric. This article covers how to infer the condition, how to confirm it functionally, and how to recover without making things worse.

What this means

The kubernetes plugin builds an in-memory record set from watch-driven state: Services, EndpointSlices, and optionally Pods. A DNS query for myservice.mynamespace.svc.cluster.local is answered from that local snapshot. The API server is only involved in keeping the snapshot current.

When the watch connection drops:

1. The snapshot freezes at its last state.
2. Existing services and endpoints continue to resolve. Latency stays normal. Error rates stay normal.
3. New services return NXDOMAIN. Endpoint changes (scale up, scale down, pod replacement) are not reflected. Deleted services may continue to resolve.
4. The longer the disconnect lasts, the further the snapshot drifts from the actual cluster.

The blast radius is limited to the Kubernetes zone, typically cluster.local. Forwarded zones (external DNS via the forward plugin) use a completely separate path and keep working. This split is one of the most useful distinguishing features: if external resolution is fine but new cluster-internal names are invisible, suspect the watch, not the network.

flowchart TD
    A[API server unreachable or overloaded] --> B[Informer watch drops]
    B --> C[Snapshot frozen at last state]
    C --> D[Existing services still resolve]
    C --> E[New services return NXDOMAIN]
    C --> F[Endpoint changes invisible]
    D --> G[Dashboards look healthy]
    E --> H[New deployments fail discovery]
    F --> I[Traffic sent to dead or missing pods]

Reconnection behavior adds a second layer of confusion. Per client-go behavior, a dropped watch is retried with exponential backoff that starts at about 800ms and is capped at 30 seconds (with jitter applied), so the stale window can last up to about a minute even after connectivity is restored.

Common causes

The RBAC case deserves emphasis. Since CoreDNS 1.11, the plugin watches EndpointSlices v1 exclusively (Endpoint and EndpointSlice v1beta watch support was removed). If the CoreDNS ServiceAccount lacks endpointslices.discovery.k8s.io permissions, the plugin cannot list or watch endpoint data at all, and logs show errors like endpointslices.discovery.k8s.io is forbidden. This commonly appears when upgrading CoreDNS past 1.11 on a cluster whose RBAC was written for the old Endpoints API.

Quick checks

These are read-only and safe to run during an incident.

# Check API request errors, broken down by HTTP status code
kubectl exec -n kube-system deploy/coredns -- \
  wget -qO- http://localhost:9153/metrics | grep coredns_kubernetes_rest_client_requests_total

# Look for code="<error>" (connection failures), code="5xx" (API trouble), code="403" (RBAC)

# Check CoreDNS logs for watch and list failures
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=200 | \
  grep -iE "watch|list|refused|forbidden|timeout"

# Test API connectivity from inside a CoreDNS pod
kubectl exec -n kube-system deploy/coredns -- \
  wget -qO- --no-check-certificate https://kubernetes.default.svc:443/healthz

# Check readiness state (proxy for plugin sync health)
kubectl exec -n kube-system deploy/coredns -- \
  wget -qO- http://localhost:8181/ready

# Baseline sanity: existing services should resolve even during a disconnect
kubectl run -it --rm dns-check --image=busybox:1.28 --restart=Never -- \
  nslookup kubernetes.default.svc.cluster.local

The most important check is the functional freshness test. Metrics can only tell you the watch is erroring; they cannot tell you whether the served data is stale. Create a throwaway Service and see if it resolves:

# Functional freshness test: create a service, then resolve it
kubectl create service clusterip dns-freshness-probe --tcp=80:80
sleep 5
kubectl run -it --rm dns-check --image=busybox:1.28 --restart=Never -- \
  nslookup dns-freshness-probe.default.svc.cluster.local
kubectl delete service dns-freshness-probe

If the probe service does not resolve within a few seconds while kubernetes.default resolves instantly, you have confirmed stale data. Clean up the probe resources when done.

How to diagnose it

1. Confirm the scope. Verify that external names resolve but recently created cluster-internal names do not. If external resolution is also broken, this is not the API disconnect pattern; check upstream forwarding instead.

2. Read the error codes. Pull coredns_kubernetes_rest_client_requests_total and group by code. code="<error>" means the HTTP client itself failed (network, connection refused, timeout). code="403" is RBAC. code="5xx" is API server-side trouble. The method and host labels tell you which operations and which API server endpoint are failing.

3. Check API request latency. coredns_kubernetes_rest_client_request_duration_seconds often rises before errors appear, as the API server degrades. A slow API delays record updates even before the watch fully breaks.

4. Check the logs. Watch disconnects, failed lists, and retry loops appear in CoreDNS logs at default verbosity. “Failed to list” errors are the clearest log-level indicator.

5. Run the freshness probe. This is the ground truth. Metrics tell you about the connection; the probe tells you about the data.

6. Check the CoreDNS version. If records stay stale even after API connectivity recovers, you may be hitting a fixed informer bug rather than a live connectivity problem. See Fixes below.

7. Check each replica independently. If you run multiple CoreDNS pods, their watches are independent. One replica can be stale while another is fresh, which produces maddening intermittent behavior depending on which pod answers. Run the freshness probe against each pod’s IP directly.

Metrics and signals to monitor

One caveat on the programming duration metric: its coverage is partial. The playbook notes it works reliably for headless_with_selector services; the cluster_ip kind was added in CoreDNS v1.14.3, and headless_without_selector remains unsupported. Do not treat silence in this metric as proof of health on older versions.

Fixes

Restore API connectivity

Fix the underlying cause: control plane overload, the NetworkPolicy or firewall rule blocking CoreDNS from reaching the API server, or the network partition. Once connectivity returns, client-go reconnects the watch automatically, though the backoff means recovery can take up to about a minute after the network heals.

Fix RBAC

If you see code=403, grant the CoreDNS ServiceAccount the permissions the plugin needs. After CoreDNS 1.11, that must include endpointslices.discovery.k8s.io. This is the single most common misconfiguration after upgrading past 1.11.

Force a fresh sync by restarting the pod

Restarting a CoreDNS pod forces a full re-list and re-sync of all objects from the API server. This is the standard way to recover a stuck or stale watch, but it is disruptive and carries two costs. First, the cache is cold on restart, so latency and upstream load briefly spike. Second, on startup the plugin waits up to 5 seconds (configurable via startup_timeout) for informer sync, and answers SERVFAIL for Kubernetes records not yet synchronized. If the API server is still unreachable, the restarted pod comes up with an empty snapshot, which is strictly worse than a stale one. Only restart after you have confirmed the API server is reachable.

Stagger restarts across replicas so at least one pod keeps serving.

Upgrade past known informer bugs

Two fixed bugs produce stale records that persist even after the API recovers:

• CoreDNS 1.7.0 fixed a tombstone handling bug in the informer cache (issue #3879) where delete events were silently dropped after protracted disconnections, so deleted services kept resolving after reconnection.
• CoreDNS 1.12.1 fixed a bug (issue #7119) where a pods verified configuration could drop pod delete deltas, leaving stale pod DNS records permanently.

If your symptoms match “stale data that survives reconnection,” check your version before treating this as a live connectivity issue.

Prevention

• Alert on the inference signal. There is no watch-broken metric, so alert on a sustained rate of code="<error>" or 5xx in coredns_kubernetes_rest_client_requests_total, corroborated before paging. The playbook’s guidance: this is TICKET-severity alone, because existing services keep working; page only when API errors combine with failed critical cluster.local resolution.
• Run a continuous freshness probe. A synthetic check that periodically creates a Service, times how long until it resolves, and deletes it catches staleness that no metric can. This is the Expert-level signal most teams skip.
• Use /ready for readiness probes, not /health. The health endpoint only checks process liveness and reports OK during a disconnect. Readiness at least waits for plugin sync at startup. Neither detects a mid-life watch failure, which is why the freshness probe matters.
• Keep RBAC in sync with CoreDNS upgrades. Treat the EndpointSlice permission as part of the upgrade checklist for any move to 1.11 or later.
• Keep CoreDNS current. The informer correctness fixes above only exist in newer versions.
• Monitor per-replica. Each pod has its own watch. Aggregate metrics mask a single stale replica.

How Netdata helps

Netdata surfaces the signals that make this failure visible despite the absence of a dedicated watch metric:

• Per-second collection of coredns_kubernetes_rest_client_requests_total broken out by code, method, and host, so the onset of <error>, 403, or 5xx responses is visible the moment the watch starts failing rather than at the next coarse scrape.
• Correlation of API client errors against zone-scoped DNS behavior: SERVFAIL and latency for cluster.local staying clean for existing names is itself the signature of stale serving, and seeing it next to the error codes confirms the pattern instead of a forwarding problem.
• DNS programming duration histograms alongside rest client latency, showing record propagation slowing before the watch breaks outright.
• Per-pod metric separation, so one stale CoreDNS replica stands out from its peers instead of disappearing into an average.
• ML anomaly detection on the error-code series, which catches the transition from a quiet baseline to a failing watch without needing a hand-tuned threshold for every cluster.

Related guides

• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS cache collapse: the cold-cache thundering herd after a rollout
• CoreDNS cache evictions: the cache is too small for the working set
• CoreDNS cache hit ratio dropping: latency and upstream load climbing together
• CoreDNS CPU throttling: CFS limits making a green dashboard lie about latency
• CoreDNS not resolving external domains: the missing catch-all forward zone
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed
• CoreDNS GC pauses adding tail latency: go_gc_duration_seconds and heap pressure
• CoreDNS goroutine count climbing: blocked upstream calls and leaks
• CoreDNS high request latency: reading P99 by zone to find the cause
• How CoreDNS actually works in production: the plugin chain mental model
• CoreDNS memory climbing: heap growth, post-GC minima, and leak detection