CoreDNS NOERROR with zero answers: the resolution failure that reports success

Your RCODE dashboards are green. NOERROR rate is healthy, SERVFAIL is near zero, health probes pass. Yet an application team reports their service cannot resolve a dependency. dig returns instantly with status: NOERROR and ANSWER SECTION: 0. No address, no error, no timeout. The resolver library treats this as “the name exists but has no records” and the application fails with a connection error that looks nothing like a DNS problem.

This is the NODATA pattern, and it is one of the most dangerous CoreDNS failure modes because nothing in a standard RCODE dashboard flags it. coredns_dns_responses_total{rcode="NOERROR"} counts a response with 47 answers and a response with zero answers identically. The metric cannot tell “resolved successfully” from “returned nothing.” Meanwhile, the cache plugin caches the empty answer as a denial entry, so the broken response persists and spreads.

This article covers how to detect NOERROR-with-zero-answers failures, which plugins produce them, and how to build alerting that does not depend on RCODE alone.

What this means

In DNS, NOERROR with an empty answer section is a legitimate response called NODATA: the name exists, but no record of the requested type exists. A correctly functioning authoritative server returns NODATA when you query a TXT record for a host that only has an A record. That is normal.

The failure mode is NODATA returned for a name and type that should have records. A service with ready endpoints returns zero A records. A hostname that resolves fine from one upstream returns empty from another. A zone that loaded 200 records yesterday answers with an empty section today. In each case the response is successful at the protocol level and broken at the operational level.

The failure cascades invisibly:

flowchart TD
  A[Plugin or upstream returns NOERROR, 0 answers] --> B[Client gets no address, no error]
  A --> C[RCODE metric increments rcode=NOERROR]
  A --> D[Cache stores entry as denial]
  C --> E[Dashboards and RCODE alerts stay green]
  D --> F[Empty answer served to later clients until TTL expiry]
  B --> G[Application fails with connection error, not DNS error]
  E --> H[Incident goes undetected]

Two properties make this worse than SERVFAIL. First, there is no error signal anywhere: not in the response code, not in the logs (the errors plugin logs errors, and NODATA is not an error), not in health or readiness probes. Second, clients do not retry NODATA the way they retry SERVFAIL or timeouts. Many resolver libraries treat it as authoritative and final. The application moves on to its connection attempt and fails there, so the error surfaces far away from DNS.

Common causes

A few of these deserve more detail:

Kubernetes service without ready endpoints. By default, the kubernetes plugin returns NOERROR with zero answers for a service whose endpoints are not ready. The service exists, so NXDOMAIN would be wrong, but there is nothing to hand back. If the deployment behind the service is crash-looping or failing readiness probes, DNS resolution “succeeds” with an empty answer and the application cannot connect. The kubernetes plugin’s ignore empty_service option changes this to return NXDOMAIN instead, which at least fails loudly.

Upstream returning SOA-less NODATA. Some appliances (global load balancers are a known offender) return NOERROR with a partial answer, for example a CNAME but no final A record, and no SOA in the authority section. CoreDNS may cache this using the CNAME TTL and serve the incomplete answer long after the upstream has fixed itself. There is no built-in metric that exposes this.

Forward plugin next_on_nodata. The forward plugin supports next_on_nodata (added in CoreDNS 1.14.4), which tries the next configured upstream when the first returns NOERROR with an empty answer section. If you have multiple forward targets and expect failover on NODATA but did not set this option on 1.14.4 or later, the first upstream’s empty answer is the final answer.

Quick checks

All read-only. Run from a pod or host with network access to CoreDNS.

# 1. Reproduce the query and inspect the full response, not just the status
dig @<coredns-ip> <failing-name> <type>
# Look at: status, ANSWER SECTION count, and whether an SOA appears in AUTHORITY

# 2. Bypass CoreDNS and ask the upstream directly
dig @<upstream-ip> <failing-name> <type> +time=2 +tries=1
# If the upstream also returns NOERROR with 0 answers, the problem is upstream

# 3. Check whether the empty answer is cached
dig @<coredns-ip> <failing-name> <type> | grep -A2 "ANSWER SECTION"
# Then query again immediately; a cached denial entry returns instantly with TTL counting down

# 4. For Kubernetes services, check the actual endpoint state
kubectl get endpoints <service> -n <namespace>
kubectl get pods -n <namespace> -l <selector> -o wide
# Empty or not-ready endpoints explain a zero-answer A response

# 5. Check response codes by plugin and zone
curl -s http://localhost:9153/metrics | grep 'coredns_dns_responses_total'
# The plugin and zone labels tell you who answered, but not whether answers were empty

# 6. Check the denial cache specifically
curl -s http://localhost:9153/metrics | grep 'coredns_cache_entries{type="denial"}'
# Growing denial entries alongside empty-answer complaints suggests cached NODATA

# 7. Check Kubernetes API connectivity if cluster.local names are affected
curl -s http://localhost:9153/metrics | grep 'coredns_kubernetes_rest_client_requests_total'
# 5xx or 403 codes indicate watch problems and possibly stale or incomplete records

How to diagnose it

1. Confirm the symptom precisely. Run dig against CoreDNS for the failing name and record type. Confirm status: NOERROR with an empty answer section. Note the authority section too: a proper NODATA from an authoritative server usually includes an SOA record. NOERROR with zero answers and no SOA is a red flag for a malformed upstream response.

2. Localize which plugin answered. Use the plugin and zone labels on coredns_dns_responses_total to see whether forward, kubernetes, or another plugin generated the response for that zone. This splits the investigation: upstream problem versus cluster-state problem versus authoritative data problem.

3. Isolate CoreDNS from the upstream. Query the upstream directly (check 2 above). If the upstream returns records and CoreDNS returns empty, the problem is in CoreDNS’s handling or caching. If the upstream also returns empty, fix the upstream or its zone data. This step determines whether you are debugging CoreDNS at all.

4. Check the cache. If the direct upstream query succeeds but CoreDNS still returns empty, the denial cache is the prime suspect. A NODATA response cached earlier is served until its TTL expires. A pod restart clears it, but treat that as confirmation, not a fix.

5. For cluster.local names, verify endpoint state and watch health. Check kubectl get endpoints for the service. If endpoints exist and are ready but DNS still returns empty, check coredns_kubernetes_rest_client_requests_total for API errors and consider the stale-watch failure mode: CoreDNS serving an outdated snapshot with all metrics green. A synthetic freshness test (create a throwaway service, measure time until it resolves) is the definitive check.

6. Review the Corefile for constructs that generate empty answers by design. Look for template blocks without an answer section (the default rcode is NOERROR with an empty answer), forward blocks where you assumed NODATA failover but did not set next_on_nodata, and any recent zone or plugin changes correlated with the onset. Check coredns_reload_failed_total to confirm the intended configuration is actually running.

Metrics and signals to monitor

The honest gap: there is no built-in CoreDNS metric that distinguishes NOERROR-with-answers from NOERROR-with-zero-answers. coredns_dns_responses_total{rcode="NOERROR"} aggregates both. Older CoreDNS versions also had bugs where responses were misreported as NOERROR in metrics entirely (NXDOMAIN reported as NOERROR via the prometheus rcode propagation, fixed in 1.9.0, and no-response-written cases reported as NOERROR, fixed in 1.8.5), so if you are on an old release, even the RCODE split is suspect.

Fixes

Upstream returns NODATA for names it should resolve

Fix the upstream’s zone data or replace the upstream. If you have multiple forward targets and want NODATA to trigger trying the next one, configure next_on_nodata in the forward block. Tradeoff: NODATA failover adds latency for every legitimately-empty response, because each one now fans out to another upstream before returning.

Kubernetes service with no ready endpoints

This is usually correct DNS behavior reflecting a real workload problem. Fix the backing deployment’s readiness. If you prefer loud failure over silent empty answers, evaluate the kubernetes plugin’s ignore empty_service option, which returns NXDOMAIN for services without ready endpoints. Tradeoff: NXDOMAIN for a temporarily endpoint-less service can confuse clients that cache negative responses, and it changes semantics for anything probing service existence versus endpoint availability.

Cached empty answers outliving the problem

The denial cache serves the NODATA response until TTL expiry. There is no targeted cache-flush command. A pod restart clears the cache but triggers a cold-cache warmup and potential thundering herd against the upstream and the Kubernetes API, so stagger restarts across replicas and avoid restarting during peak load. Longer term, review denial cache TTL caps in the cache plugin so a transient upstream NODATA cannot poison answers for an extended period.

Template or zone misconfiguration

Correct the template block to return an answer or an explicit error rcode, and fix zone data load failures. Watch coredns_reload_failed_total after every Corefile change; a failed reload means the old configuration is still running while you believe the fix is live.

SOA-less partial responses from appliances

If an upstream appliance returns NOERROR with a CNAME but no final records and no SOA, CoreDNS can cache the incomplete response for the CNAME’s TTL. The durable fix is at the appliance or by fronting it with a resolver that normalizes responses. Short-term, capping cache TTLs limits how long the broken answer persists.

Prevention

• Deploy synthetic resolution checks with answer-count assertions. Periodically dig a small set of known-good names per zone (one cluster.local service, one external name, one name per authoritative zone) and alert when a query returns NOERROR with fewer answers than expected. This is the only detection mechanism that catches the failure directly.
• Alert on denial cache growth, not just RCODE. A rising coredns_cache_entries{type="denial"} trend relative to success entries is an early indicator that empty answers are being generated and cached.
• Test freshness, not just liveness. Neither /health (port 8080) nor /ready (port 8181) tests actual DNS resolution or record freshness. A synthetic create-service-then-resolve check catches stale-watch scenarios that look identical to healthy operation in metrics.
• Monitor per-upstream and per-plugin breakdowns. Dashboard the to label on proxy metrics and the plugin label on responses so a single misbehaving upstream or plugin is visible before it dominates aggregate numbers.
• Keep CoreDNS current. Metric accuracy bugs that inflated NOERROR counts were fixed in past releases; running an old version makes even the signals you do have less trustworthy.
• Review Corefile constructs that produce empty answers intentionally. Any template block without an answer, any forward chain without next_on_nodata, and any zone whose data source can partially load deserves a comment in the file and a synthetic check.

How Netdata helps

• Netdata collects the CoreDNS Prometheus endpoint on port 9153, including coredns_dns_responses_total broken out by rcode, plugin, and zone, so you can see which plugin is answering a zone during an incident window without hand-rolling queries.
• Cache composition charts (coredns_cache_entries split by success and denial) make denial-cache growth visible as a trend, which is the closest metric-based proxy for spreading NODATA answers.
• Kubernetes integration metrics (coredns_kubernetes_rest_client_requests_total by status code, DNS programming duration) sit next to DNS metrics on the same dashboard, so stale-watch and partial-sync causes are correlatable with resolution complaints in one view.
• Per-second granularity on query rate and latency helps you correlate the exact moment an upstream or Corefile change occurred with the onset of empty-answer reports, even when RCODE ratios never move.

Related guides

• How CoreDNS actually works in production: the plugin chain mental model
• CoreDNS monitoring checklist: the signals every production resolver needs
• CoreDNS monitoring maturity model: from survival to expert
• CoreDNS returning SERVFAIL: the resolver is failing queries and what to check first