CoreDNS CPU throttling: CFS limits making a green dashboard lie about latency

Applications are reporting intermittent DNS timeouts and multi-hundred-millisecond resolution spikes. You open the CoreDNS dashboard and everything looks fine: coredns_dns_request_duration_seconds shows sub-millisecond cache hits, SERVFAIL is zero, QPS is normal, pods are healthy. Both things are true at the same time, and that is exactly the problem.

When a CoreDNS pod has a tight CPU limit, the Linux CFS scheduler throttles the container: the process is runnable but the kernel refuses to schedule it until the next quota period. That stall is real latency for the client, but it never appears in any CoreDNS metric, because CoreDNS only measures time spent actively processing a query. Time spent waiting for CPU does not exist as far as coredns_dns_request_duration_seconds is concerned.

This is one of the nastier CoreDNS failure modes: application-observed DNS latency can run 10 to 100 times the number CoreDNS reports, while every CoreDNS-native signal stays green. The fix is almost never in CoreDNS itself. It is in the pod’s CPU limit.

What this means

Kubernetes translates resources.limits.cpu into a CFS bandwidth quota on the pod’s cgroup. The CFS scheduler enforces that quota per accounting period, which is 100ms by default. A limit of 100m means the container may consume 10ms of CPU time per 100ms period. If CoreDNS burns through its 10ms in the first 15ms of a period, every runnable thread in the container is frozen for the remaining 85ms. Queries that arrive during the freeze wait.

Two properties make this treacherous:

1. Throttling is per-period, not averaged. A pod can show 5-15% average CPU utilization in kubectl top or Grafana and still be throttled thousands of times per minute, because averages hide what happens inside each 100ms window. Bursty DNS load is the worst case: a burst exhausts the quota early in the period, then everything stalls.
2. CoreDNS cannot see it. coredns_dns_request_duration_seconds measures handler execution time. Scheduling delay sits outside the handler. The same blind spot applies to kernel socket buffer wait time and network transit, which is why the rule of thumb is to never trust CoreDNS metrics alone for application-observed latency.

flowchart LR
  A[Client DNS query] --> B[Kernel delivers packet to socket]
  B --> C{CoreDNS runnable?}
  C -- "quota exhausted: frozen until next 100ms period" --> D[Throttled wait - invisible to CoreDNS metrics]
  C -- "scheduled" --> E[Plugin chain processing - measured by coredns_dns_request_duration_seconds]
  D --> E
  E --> F[Response sent]
  D -.-> G[cpu.stat nr_throttled and throttled_time - only place the stall is recorded]

The client experience is B-to-F. The CoreDNS dashboard shows only E. The gap between them lives in the cgroup’s cpu.stat counters and nowhere else.

Common causes

Quick checks

All of these are read-only.

# 1. What is the CoreDNS CPU limit?
kubectl get deploy coredns -n kube-system -o jsonpath='{.spec.template.spec.containers[*].resources}'

# 2. Average CPU (expect this to look deceptively fine)
kubectl top pod -n kube-system -l k8s-app=kube-dns

# 3. Read the throttling counters directly (cgroup v1 path)
kubectl exec -n kube-system <coredns-pod> -- cat /sys/fs/cgroup/cpu/cpu.stat
# Watch nr_throttled and throttled_time (nanoseconds).
# On cgroup v2 the file is /sys/fs/cgroup/cpu.stat
# and the fields are nr_throttled and throttled_usec (microseconds).

# 4. Sample twice to get a throttling rate
kubectl exec -n kube-system <coredns-pod> -- cat /sys/fs/cgroup/cpu/cpu.stat
sleep 60
kubectl exec -n kube-system <coredns-pod> -- cat /sys/fs/cgroup/cpu/cpu.stat
# nr_throttled increasing by hundreds+ per minute under load is the smoking gun.

# 5. Compare CoreDNS-reported latency with client-observed latency
kubectl exec -n kube-system <coredns-pod> -- \
  wget -qO- http://localhost:9153/metrics | grep '^coredns_dns_request_duration_seconds'
dig @<coredns-cluster-ip> kubernetes.default.svc.cluster.local +stats
# Run the dig from an application pod, repeatedly. "Query time" well above
# the CoreDNS histogram while the histogram stays flat confirms the gap.

# 6. Check CoreDNS's health self-check latency as a corroborating signal
kubectl exec -n kube-system <coredns-pod> -- \
  wget -qO- http://localhost:9153/metrics | grep 'coredns_health_request_duration_seconds'
# Rising self-check latency indicates the process is CPU-starved.
# <!-- TODO: verify this metric is exported by the health plugin; if it is not,
# measure :8080/health response time externally instead -->

If you have cAdvisor metrics in Prometheus, the cleanest aggregate views are the fraction of periods throttled, rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m]), and the average throttled time per period, rate(container_cpu_cfs_throttled_seconds_total[5m]) / rate(container_cpu_cfs_periods_total[5m]) (multiply by 1000 for milliseconds per 100ms period). A throttled-period fraction sustained above roughly 0.05 is a real problem for a latency-sensitive resolver.

How to diagnose it

1. Establish the gap. From an application pod, time DNS resolution in a loop and capture the distribution. Compare against CoreDNS’s coredns_dns_request_duration_seconds percentiles for the same window. If app-observed latency is 10-100x the CoreDNS number, you are looking at delay outside the handler: throttling, socket buffer wait, or network path. Throttling is the most common of the three for a CPU-limited pod.
2. Confirm with cgroup counters. Pull cpu.stat twice, 60 seconds apart, during a latency spike. Compute deltas for nr_throttled and throttled_time (or throttled_usec on cgroup v2). If throttled time per minute is a large fraction of wall time, the pod is spending meaningful portions of each 100ms period frozen.
3. Check the pattern against load. Correlate throttling deltas with QPS (coredns_dns_requests_total). Throttling that tracks query bursts means the limit is too tight for peak, not that something is pathologically spinning.
4. Rule out lookalikes. If cpu.stat shows no throttling, the same “green CoreDNS, slow clients” symptom comes from UDP buffer drops (/proc/net/snmp Udp RcvbufErrors), conntrack exhaustion (nf_conntrack: table full in dmesg), or the A/AAAA race producing 5-second client timeouts. Those have different fixes. See CoreDNS monitoring checklist: the signals every production resolver needs.
5. Check for demand-side amplifiers. A high NXDOMAIN ratio and QPS 4-6x your logical lookup rate means ndots:5 amplification is multiplying CPU demand. Fixing demand can be as effective as raising the limit.

Metrics and signals to monitor

Fixes

Right-size the CPU limit

This is the actual fix. Raise the limit until throttling stops under peak load, not average load. As a sizing rule, keep sustained CPU below about 60% of the limit, because CFS throttling becomes frequent enough to damage tail latency well before you hit 100%. A typical pattern: CoreDNS at 97m against a 100m limit throttles constantly; moving to a 500m limit eliminates throttling entirely.

Consider dropping the limit and keeping only a request

For a latency-critical cluster service, CPU limits buy isolation at the cost of exactly this failure mode. Setting a CPU request (for scheduling and QoS) without a limit lets CoreDNS absorb bursts using spare node capacity. The tradeoff: a runaway CoreDNS can now contend with neighbors, so pair this with node-level monitoring and keep memory limits in place (memory overcommit is far more dangerous than CPU overcommit).

Reduce the demand, not just raise the supply

If ndots:5 amplification is multiplying your QPS, you are paying CFS quota for search-domain misses. Setting ndots: "1" (or 2) in dnsConfig for workloads that mostly resolve external names, or fully qualifying names with a trailing dot in application config, cuts CoreDNS CPU demand directly. NodeLocal DNSCache offloads repeat queries from CoreDNS entirely, at the cost of an extra layer to monitor.

Isolate CoreDNS from noisy neighbors

Node-level contention stacks on top of the cgroup quota. If CoreDNS shares nodes with CPU-hungry batch workloads, taints, dedicated nodes, or at least guaranteed-QoS placement reduce the interference that turns marginal quota into constant throttling.

Do not reach for a pod restart as remediation. A restart clears nothing here; the limit and the load are unchanged, and the pod starts throttling again as soon as traffic returns.

Prevention

• Alert on throttling, not just utilization. Alert on throttled fraction or nr_throttled rate for CoreDNS pods. Average CPU alerts will not fire for this failure mode, ever.
• Monitor application-observed DNS latency alongside CoreDNS-reported latency, and alert on divergence between them. That divergence is the catch-all for every delay CoreDNS cannot see: throttling, socket buffer waits, conntrack drops.
• Size for peak, then verify. Load-test DNS under realistic burst patterns (including ndots amplification) and watch cpu.stat during the test, not just average CPU.
• Include health-check response time on the CoreDNS dashboard (via the metric above if exported, or by timing the :8080/health endpoint). It is a cheap proxy for CPU starvation.
• Revisit limits after growth events. Cluster growth, new workloads, and HPA events raise QPS. A limit that was fine at 100 pods throttles at 400. This failure archetype and its neighbors are covered in How CoreDNS actually works in production: the plugin chain mental model.

How Netdata helps

• Per-second CPU and cgroup throttling visibility for containers, so the 100ms-period stall pattern is visible instead of being averaged away.
• CoreDNS latency, QPS, SERVFAIL, and cache metrics collected from the :9153/metrics endpoint, letting you put coredns_dns_request_duration_seconds next to container throttling in one view and see the divergence directly.
• Go runtime metrics (goroutines, GC) to separate “throttled by quota” from “slow handler” when latency does rise inside CoreDNS.
• Node-level CPU saturation and per-pod context, so you can tell a tight-limit problem from a noisy-neighbor problem.
• Anomaly detection on throttling counters, which catches the transition from “occasionally throttled” to “throttled every period” before clients start timing out.

Related guides

• CoreDNS monitoring checklist: the signals every production resolver needs
• How CoreDNS actually works in production: the plugin chain mental model
• CoreDNS monitoring maturity model: from survival to expert
• CoreDNS cache collapse: the cold-cache thundering herd after a rollout
• CoreDNS cache hit ratio dropping: latency and upstream load climbing together
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed
• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken