$ guides / coredns / coredns-forward-max-concurrent-rejects ▌

Operations Guides

CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed

coredns_forward_max_concurrent_rejects_total is incrementing and clients are getting REFUSED responses for forwarded queries. Something is piling up between CoreDNS and your upstreams, and the rejects are the valve letting pressure escape.

This counter only increments when the number of in-flight forwarded queries hits the max_concurrent cap you configured. On current CoreDNS releases (1.8.1 and later) every rejected query gets a REFUSED response, not SERVFAIL; releases before 1.8.1 returned SERVFAIL instead. That distinction matters for triage: REFUSED from this path is a capacity signal, not a resolution failure, and unlike NXDOMAIN it is not subject to negative caching, so it will not linger in client caches after the pressure clears.

The rejects are the symptom of one of two underlying problems: your upstreams are slow and queries are accumulating while waiting for answers, or your query volume genuinely exceeds what the configured limit allows. The rest of this guide is about telling those apart and fixing the right one.

What this means

The forward plugin hands each forwarded query a goroutine and a slot in the upstream connection pool. Under normal conditions, a query enters the plugin, waits a few milliseconds for an upstream response, and completes. The in-flight count stays low because throughput is high relative to latency.

When upstream latency rises, each query occupies its in-flight slot longer. Little’s law applies directly: concurrent in-flight queries equals forwarded QPS times average upstream latency. At 5,000 forwarded QPS with 10ms upstream latency, you hold about 50 concurrent queries. If that upstream latency degrades to 500ms, the same QPS now needs 2,500 concurrent slots. If max_concurrent is 1,000, the excess 1,500 queries per second get REFUSED and the counter climbs.

This is intentional backpressure. Without the cap, those queries would accumulate as blocked goroutines, each holding stack memory and request buffers, until the pod exhausts memory and gets OOM killed. The cap trades a clean, fast REFUSED for a slow memory blowout. Your job is to figure out why the queue filled.

flowchart LR
  Q[Forwarded query] --> C{In-flight count below max_concurrent?}
  C -->|yes| U[Send to upstream, wait for response]
  U --> R[Return answer to client]
  C -->|no| X[Increment rejects counter, return REFUSED]
  U -.->|upstream slow| P[Queries pile up in-flight]
  P --> C

max_concurrent defaults to unlimited. If this counter is incrementing, someone set the limit explicitly, either in your Corefile or in a platform default ConfigMap. Find out what it is set to and who chose that number before deciding whether the limit or the upstream is the problem.

Common causes

Cause	What it looks like	First thing to check
Slow upstream DNS	Rejects climb alongside rising `coredns_forward_request_duration_seconds`; goroutine count grows	Per-upstream latency via the `to` label
Query volume spike	Rejects during a traffic burst, cold cache after a rollout, or a client retry storm; upstream latency normal	Forwarded QPS vs the configured limit
Limit set too low	Rejects at normal traffic levels with healthy upstream latency; started right after a config change	The Corefile value vs QPS times latency
Single degraded upstream	One upstream slow, others fine; rejects appear only when load balancing routes to the slow one	`coredns_forward_healthcheck_failures_total` per upstream
DDoS or amplification flood	Query rate far above baseline, unusual query types (ANY), rejects saturated	Query rate and type distribution

Quick checks

All read-only. Replace the metrics address if you scrape a different endpoint.

# 1. Confirm the rejects counter and its current value
curl -s http://localhost:9153/metrics | grep 'coredns_forward_max_concurrent_rejects_total'

# 2. Check per-upstream latency: which upstream is slow?
curl -s http://localhost:9153/metrics | grep 'coredns_forward_request_duration_seconds'

# 3. Check per-upstream health check failures
curl -s http://localhost:9153/metrics | grep 'coredns_forward_healthcheck_failures_total'

# 4. Check goroutine count (baseline is typically 20-50)
curl -s http://localhost:9153/metrics | grep '^go_goroutines'

# 5. Check the REFUSED response rate
curl -s http://localhost:9153/metrics | grep 'coredns_dns_responses_total' | grep 'REFUSED'

# 6. Check cache hit ratio inputs (a cold cache amplifies forwarded load)
curl -s http://localhost:9153/metrics | grep -E 'coredns_cache_(hits|requests)_total'

Pull the current Corefile and look at the forward stanza. In Kubernetes: kubectl get cm -n kube-system coredns -o yaml. Note the max_concurrent value, the upstream list, and whether a recent change introduced or altered the limit.

If you suspect a specific upstream, test it directly from the CoreDNS network path:

# Measure upstream response time with a real DNS query
dig @<upstream_ip> example.com +time=2 +tries=1 +stats

A slow or timed-out dig to one upstream while others answer fast confirms the per-upstream latency signal.

How to diagnose it

Establish the reject rate, not just the counter. The counter is cumulative; what matters is whether it is incrementing right now and how fast. Sample it twice a minute apart, or use your metrics backend. A flat counter with a large value is history, possibly from a past incident. A rising counter is a live problem.
Correlate rejects with upstream latency. Pull coredns_forward_request_duration_seconds broken down by the to label. If P99 upstream latency rose sharply at the same time rejects started, you have a slow-upstream problem: the limit is doing its job and the fix is at the upstream or the network path. If upstream latency is flat and healthy, the limit is undersized for your current volume.
Check whether this is a volume event. Look at forwarded query rate and cache hit ratio. Rejects that begin exactly at a rollout, a restart, or a ConfigMap reload point to a cold-cache flood: hit ratio dropped, every query forwards, the in-flight count spikes, and the cap trips. These events are transient and self-correct as the cache warms, typically within minutes. A retry storm from a misbehaving client looks similar but does not self-correct.
Check goroutines and memory. go_goroutines well above baseline (typically 20-50 at idle) alongside rejects tells you queries are genuinely accumulating, which supports the slow-upstream theory. If goroutines are near baseline while rejects fire, the pileup is short-lived, more consistent with a volume spike than sustained upstream drag.
Isolate the upstream. If latency is the cause, the to label tells you which upstream. Compare it against its peers and against your dig test. Also check coredns_forward_healthcheck_failures_total for that upstream: an upstream that passes health checks but answers real queries slowly will not be failed over automatically.
Decide: fix the upstream path or fix the limit. Slow upstream means replace, remove, or investigate the upstream and its network path. Healthy upstream with a tripped cap means recalculate the limit or reduce forwarded volume (cache tuning, ndots amplification). Do not raise the cap to absorb a slow upstream: that converts clean REFUSEDs into goroutine accumulation and moves you toward an OOM kill.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`coredns_forward_max_concurrent_rejects_total`	The reject counter itself; rate of change shows live backpressure	Any sustained nonzero rate; rejects above roughly 1% of forwarded queries
`coredns_forward_request_duration_seconds{to=...}`	Per-upstream latency; the primary cause signal	P99 above 250ms sustained, or a sharp jump from baseline
`go_goroutines`	Confirms real query accumulation vs transient bursts	Sustained above 2x baseline, or a trend that never returns to baseline
`coredns_dns_responses_total{rcode="REFUSED"}`	Client-visible impact of the rejects	Nonzero sustained rate during normal operations
`coredns_cache_hits_total` / `coredns_cache_requests_total`	A dropping hit ratio means more queries forward, raising in-flight pressure	Sudden drop after a restart, reload, or eviction event
`coredns_forward_healthcheck_failures_total{to=...}`	Identifies an upstream that is failing or flapping	Sustained failures for one upstream
`process_resident_memory_bytes` vs container limit	The failure mode the cap is protecting you from	RSS trending toward 80% of the pod memory limit

For alerting: page on a composite of sustained reject rate plus elevated REFUSED response rate plus failed resolution of a critical name. Alert at ticket level on any sustained nonzero reject rate alone. Transient trips during cold-cache floods and batch bursts are expected behavior, so a raw “counter > 0” alert will page you for non-events.

Fixes

Slow upstream

Remove or replace the degraded upstream in the Corefile forward stanza. If you have multiple upstreams and one is slow, dropping it temporarily restores capacity immediately. Investigate the network path: rate limiting by a cloud provider resolver, a firewall or inspection appliance adding latency, or congestion. Verify with the dig test above from the same network path CoreDNS uses. Do not raise max_concurrent as the primary response here; you would be trading fast rejections for memory growth.

Limit undersized for legitimate volume

Resize from measured values, not guesses. Expected concurrent queries equals forwarded QPS times average upstream latency in seconds; set max_concurrent to at least 3x that to absorb bursts and upstream slowdowns. At 10,000 forwarded QPS with 10ms average upstream latency, that is 100 concurrent queries, so a limit of 300 or more. Many production Kubernetes ConfigMaps carry max_concurrent 1000 as a de facto default; that number is only right if your QPS-times-latency product is well below it.

Volume spikes from cold cache

If rejects track rollouts or restarts, reduce the blast radius of cache cold-start: stagger CoreDNS rollouts (maxUnavailable=1 or a PodDisruptionBudget), and confirm cache sizing and TTLs are not forcing unnecessary upstream traffic. The rejects during warmup are the cap working as designed; the fix is preventing cluster-wide cache resets.

Client retry storms or query amplification

If a specific workload is flooding forwarded queries, fix it at the source. In Kubernetes, check whether ndots:5 search-domain expansion is amplifying external lookups several-fold, and check for applications retrying failed lookups aggressively. Reducing amplified volume lowers in-flight pressure without touching the limit.

Prevention

Size the limit from data. Compute forwarded QPS times upstream latency from your own metrics and set max_concurrent to at least 3x. Revisit after traffic growth or upstream changes.
Alert on rate, not counter. Sustained reject rate corroborated by REFUSED rate is actionable. A nonzero counter after a one-time flood is not.
Watch upstream latency independently. Per-upstream P99 trending upward is your early warning before rejects start. Do not wait for the cap to trip.
Keep the cap set. An unlimited forward plugin under a slow-upstream event grows goroutines until OOM. The cap existing and tripping occasionally is healthier than the cap not existing.
Stagger restarts and rollouts. Prevent cluster-wide cold-cache floods so the cap is reserved for genuine anomalies.

How Netdata helps

Netdata collects coredns_forward_max_concurrent_rejects_total per CoreDNS instance, so you can see which pod is rejecting rather than a blended average that hides a single degraded replica.
Per-upstream forward latency and health check failures are charted alongside the reject counter, letting you confirm or rule out the slow-upstream cause in one view.
Goroutine count and memory charts next to the reject rate distinguish genuine query accumulation from transient bursts.
Cache hit ratio and REFUSED response rate on the same dashboard expose cold-cache events and client-visible impact without manual correlation.
Anomaly detection on upstream latency surfaces the slow creep toward the cap before rejects begin.