CoreDNS 403 from the Kubernetes API: RBAC misconfiguration breaking cluster DNS

The symptom usually arrives in one of two forms. Either CoreDNS pods never become ready after an install or upgrade, stuck at 0/1 while kubectl shows no crash and no obvious error, or cluster DNS suddenly returns SERVFAIL for cluster.local names while external resolution keeps working. In both cases, the smoking gun is the same: coredns_kubernetes_rest_client_requests_total{code="403"} is incrementing.

A 403 from the Kubernetes API is not a transient error. It means the API server received the request, authenticated the caller, and refused it because the CoreDNS ServiceAccount lacks the RBAC permissions the kubernetes plugin needs to list and watch Services, Endpoints, and EndpointSlices. Without those watches, the plugin cannot build or maintain its in-memory record set, so it cannot answer queries for the cluster zone.

This signal is binary-actionable: any 403 is a misconfiguration, full stop. There is no legitimate steady state in which the API server 403s CoreDNS. The fix is always in RBAC, not in CoreDNS itself, and restarting the pod before fixing RBAC accomplishes nothing.

The timing is a strong hint. This failure almost always surfaces right after a CoreDNS install, a CoreDNS or Kubernetes upgrade, or a change to the system:coredns ClusterRole or its binding. The classic trap is the CoreDNS 1.8.1 boundary: from that release on, the kubernetes plugin requires list and watch access to EndpointSlices on clusters running Kubernetes 1.19 or later. Upgrading CoreDNS across that boundary without updating the ClusterRole produces exactly this failure.

What this means

The kubernetes plugin does not query the API server per DNS request. It maintains persistent watches (informers) over the resources that make up cluster DNS: Services, Endpoints or EndpointSlices, and optionally Pods and namespaces. From that watch stream it builds an in-memory snapshot and answers cluster.local queries from it.

When the API server answers the plugin’s list or watch calls with 403, the informer can never complete its initial sync. CoreDNS logs a warning about starting with an unsynced Kubernetes API, and the kubernetes plugin returns SERVFAIL for queries in the cluster zone. Two consequences follow:

1. Readiness never passes. The ready plugin (port 8181) waits for the kubernetes plugin to sync before reporting ready. With RBAC broken, the pod stays not-ready indefinitely. If your readiness probe points at /ready (as it should), the pod is pulled from the kube-dns Service endpoints. If all replicas are affected, cluster DNS is down even though the processes are running and /health returns 200.
2. Cluster zone SERVFAILs, external zones keep working. The forward plugin does not use the Kubernetes API, so external name resolution is unaffected. This split (external fine, internal broken) is itself a diagnostic signal: it isolates the problem to the kubernetes plugin rather than to upstreams or the network path.

Be precise about what a 403 is not. A 403 is deterministic authorization denial, not API overload (which looks like 5xx or timeouts) and not a network problem (which looks like connection errors with no HTTP status at all). The code label on the metric tells you which of these three worlds you are in, and the remediation for each is completely different.

flowchart TD
  A[kubernetes plugin cannot sync] --> B{Check coredns_kubernetes_rest_client_requests_total by code}
  B -->|code=403| C[RBAC denial: fix ClusterRole or binding]
  B -->|code=5xx| D[API server overload or etcd lag: fix control plane]
  B -->|no HTTP code / conn errors| E[Network path to API server: NetworkPolicy, routing]
  C --> F[Missing endpointslices rule?]
  C --> G[Binding subject namespace wrong?]
  C --> H[Role changed or reverted by tooling?]
  F --> I[Apply canonical RBAC, restart pods]
  G --> I
  H --> I

Common causes

The EndpointSlices case deserves emphasis because it is the most common in the wild. Before CoreDNS 1.8.1, the plugin watched the older Endpoints API only, and the ClusterRole from that era contains no discovery.k8s.io rule. On Kubernetes 1.19 and later, newer CoreDNS versions watch EndpointSlices instead. Any upgrade path that bumps CoreDNS (kubeadm upgrades, EKS addon updates, Helm chart bumps) without touching RBAC walks straight into this. Documented production outages exist from exactly this move, for example upgrading CoreDNS 1.8.0 to 1.8.4 on EKS where the system:coredns ClusterRole had never been updated.

Quick checks

All of these are read-only. One caveat: the CoreDNS container image has no shell and no wget or curl, so kubectl exec is not available. Fetch metrics and readiness through the API server’s pod proxy instead.

# Pick a CoreDNS pod once, reuse for checks 1-2
POD=$(kubectl get pod -n kube-system -l k8s-app=kube-dns \
  -o jsonpath='{.items[0].metadata.name}')

# 1. Confirm the 403s and see which codes are present
kubectl get --raw "/api/v1/namespaces/kube-system/pods/${POD}:9153/proxy/metrics" \
  | grep coredns_kubernetes_rest_client_requests_total
# code="403" incrementing = RBAC. code="5xx" = API overload. No 4xx/5xx but sync failing = network.

# 2. Check readiness (should be failing if RBAC is broken)
kubectl get --raw "/api/v1/namespaces/kube-system/pods/${POD}:8181/proxy/ready"
# Non-200 / empty means plugins have not synced.

# 3. Read the logs; the forbidden resource is named explicitly
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100 \
  | grep -iE "forbidden|cannot list|cannot watch|unsynced"
# Typical line: "system:serviceaccount:kube-system:coredns cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\""

# 4. Inspect the ClusterRole
kubectl get clusterrole system:coredns -o yaml
# Expect list/watch on endpoints, services, pods, namespaces (apiGroup "")
# AND endpointslices (apiGroup discovery.k8s.io) for CoreDNS >= 1.8.1 on k8s >= 1.19.

# 5. Inspect the binding subject
kubectl get clusterrolebinding system:coredns -o yaml
# The subject's namespace and name must match the actual CoreDNS ServiceAccount.

# 6. Verify effective permissions directly
kubectl auth can-i list endpointslices.discovery.k8s.io \
  --as=system:serviceaccount:kube-system:coredns
kubectl auth can-i watch services \
  --as=system:serviceaccount:kube-system:coredns

# 7. Functional check: does the cluster zone resolve?
kubectl run -it --rm --restart=Never dns-check --image=busybox:1.28 \
  -- nslookup kubernetes.default.svc.cluster.local
# SERVFAIL while external names resolve confirms kubernetes-plugin impact.

Note on check 7: some hardened clusters restrict ad-hoc pods in the default namespace. Run it from an existing pod if needed. The point is to prove the split between internal and external resolution.

How to diagnose it

1. Confirm the failure domain. Check SERVFAILs via coredns_dns_responses_total{rcode="SERVFAIL"} and note the plugin and zone labels. SERVFAIL from the kubernetes plugin on the cluster.local zone, with external names resolving, points at the plugin’s API dependency, not at forward or upstreams.
2. Read the code label. Pull coredns_kubernetes_rest_client_requests_total and group by code. Any 403 is RBAC. If instead you see 5xx, stop following this guide and investigate API server health. If you see connection errors with no HTTP codes, investigate the network path (NetworkPolicy, routing, API server endpoint).
3. Identify the forbidden resource. The CoreDNS logs name it: the reflector errors say which resource and API group were denied (endpointslices in discovery.k8s.io, or endpoints/services/pods/namespaces in the core group). This tells you which rule is missing.
4. Verify the identity being denied. The log line includes the full system:serviceaccount:<namespace>:<name> identity. Check that it matches the ServiceAccount the CoreDNS pods actually run as (kubectl get deploy coredns -n kube-system -o jsonpath='{.spec.template.spec.serviceAccountName}'). A Helm release into a different namespace can leave the binding subject pointing at the wrong ServiceAccount while the ClusterRole itself is perfect.
5. Diff the ClusterRole against the canonical manifest. Compare the live role to the one shipped with the CoreDNS deployment manifests for your CoreDNS version. Pay attention to the verbs: the plugin needs list and watch, and a role that grants only get will 403 exactly the same way.
6. Check when it broke. Correlate the start of 403s with cluster events: CoreDNS image bumps, Kubernetes upgrades, GitOps syncs, RBAC changes. The answer is almost always one of those, and knowing which one tells you which mechanism will re-break it if you only patch the live object.

Metrics and signals to monitor

One subtlety: there is no binary “watch broken” metric. The code label on the REST client counter is how you infer sync health. And /health (port 8080) only checks process liveness; a pod can be perfectly “healthy” there while every cluster.local query SERVFAILs. This is why the readiness probe should point at /ready.

Fixes

Missing or stale ClusterRole rules

Apply the canonical RBAC for your CoreDNS version. For CoreDNS 1.8.1 and later on Kubernetes 1.19+, the ClusterRole must grant list and watch on endpoints, services, pods, and namespaces in the core API group, plus endpointslices in discovery.k8s.io. The reference manifest lives in the CoreDNS deployment repository (the coredns.yaml template used by kubeadm-style installs) and the kubernetes plugin documentation describes what the plugin watches.

Prefer patching the manifest your tooling owns (Helm values, kustomize overlay, addon manager config) over hand-editing the live object, or the next reconciliation will revert your fix and the 403s will return at the worst possible time.

Wrong ClusterRoleBinding subject

If the logs show an identity that does not exist or lives in the wrong namespace, fix the binding’s subject to reference the actual ServiceAccount the CoreDNS pods run as. This is the classic Helm trap: the chart creates a correctly-shaped ClusterRole but binds it to a ServiceAccount name or namespace that does not match the release. The ClusterRole looks fine on inspection and the 403s persist, which is exactly why you check the identity in the log line before touching the role.

After the RBAC fix

RBAC changes take effect immediately, but the informers inside a running CoreDNS may be sitting in long backoff loops from repeated denials. The reliable path is to roll the pods after fixing RBAC:

# Roll CoreDNS after fixing RBAC so informers start a fresh list/watch
kubectl rollout restart deployment/coredns -n kube-system
kubectl rollout status deployment/coredns -n kube-system

Watch readiness come up, then re-run the functional check (nslookup kubernetes.default.svc.cluster.local) and confirm code="403" has stopped incrementing. rollout restart respects the deployment’s rolling strategy, but if all replicas are currently not-ready, DNS is already down and the restart restores service rather than risking it. If only some replicas were affected and partial capacity matters, verify each new pod is ready before the old one terminates.

What not to do: do not restart pods before fixing RBAC (they will fail identically), do not widen the role to cluster-admin as a shortcut (the plugin needs five resource types, not the keys to the cluster), and do not treat this as a CoreDNS bug.

Prevention

• Pin RBAC to the CoreDNS version in the same change. Every CoreDNS upgrade should carry its RBAC diff in the same commit or chart bump. The 1.8.1 EndpointSlices requirement is the canonical example of why these must move together.
• Alert on any 403, not on a threshold. coredns_kubernetes_rest_client_requests_total{code="403"} going nonzero is a page-worthy config regression, not a capacity signal. It should fire in staging the moment a bad upgrade lands, long before production.
• Alert on readiness duration, not just pod restarts. Broken RBAC produces a running, healthy-looking, never-ready pod. “Alive but not ready for > 60s” catches this; restart-count alerts do not, because nothing restarts.
• Check binding subjects in CI for Helm installs. Render the chart and assert the ClusterRoleBinding subject matches the ServiceAccount the deployment will actually use, in the actual release namespace.
• Verify post-upgrade, automatically. After any control-plane or CoreDNS upgrade, run the functional check (resolve a known cluster name from a pod) and confirm readiness across all replicas before declaring the upgrade done.

How Netdata helps

• Netdata charts coredns_kubernetes_rest_client_requests_total split by code, so a 403 series appearing is immediately visible against the healthy 200 baseline, and you can tell RBAC denial apart from 5xx API overload at a glance.
• Correlating API 403s with SERVFAIL responses by plugin and zone on the same dashboard confirms the impact path (kubernetes plugin, cluster.local only) without log diving.
• Per-pod readiness and restart state alongside the metrics shows the “running but never ready” shape that pure pod-health monitoring misses.
• Alerts can fire on any nonzero 403 rate and on prolonged not-ready state, catching the regression at upgrade time rather than when developers report broken service discovery.

Related guides

• How CoreDNS actually works in production: the plugin chain mental model
• CoreDNS high request latency: reading P99 by zone to find the cause
• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS not resolving external domains: the missing catch-all forward zone
• CoreDNS cache collapse: the cold-cache thundering herd after a rollout
• CoreDNS cache hit ratio dropping: latency and upstream load climbing together
• CoreDNS cache evictions: the cache is too small for the working set
• CoreDNS CPU throttling: CFS limits making a green dashboard lie about latency
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed
• CoreDNS GC pauses adding tail latency: go_gc_duration_seconds and heap pressure
• CoreDNS goroutine count climbing: blocked upstream calls and leaks
• CoreDNS memory climbing: heap growth, post-GC minima, and leak detection