CoreDNS NXDOMAIN flood from one source: DGA malware and domain enumeration

Your CoreDNS dashboards are green: SERVFAIL is zero, latency is fine, cache hit ratio looks normal. But the NXDOMAIN rate has jumped, and when you pull query logs, one source IP is responsible for thousands of NXDOMAIN responses per hour, querying names that do not exist.

This is a security signal, not a CoreDNS health signal. A single source generating a sustained flood of NXDOMAIN responses is doing one of three things: cycling randomly generated domains looking for a command-and-control server (DGA malware in a compromised pod), probing which service names exist in your cluster (reconnaissance), or misbehaving in a way that merely looks like an attack (a config typo hammering a name that was never created).

The hard part is that CoreDNS gives you almost no help here out of the box. There is no per-source-IP metric. coredns_dns_responses_total{rcode="NXDOMAIN"} aggregates across all clients, and the NXDOMAIN rate in Kubernetes is already high and noisy because search-domain expansion generates NXDOMAINs constantly. Detection is log-based, and triage is about the shape of the queried names, not the count alone.

What this means

NXDOMAIN is a normal response code. It means “this name does not exist,” and in Kubernetes it is expected: with the default ndots:5 resolver configuration, a lookup for an external name is tried against cluster-internal suffixes first, and each suffix miss returns NXDOMAIN. Baseline NXDOMAIN ratios of 20 to 60 percent of total responses are common and healthy.

The anomaly is not NXDOMAIN itself. It is the concentration: one client IP responsible for a disproportionate share of NXDOMAIN responses, sustained over time. A reasonable starting flag threshold is a single source generating more than 100 NXDOMAIN responses per minute, but the right number depends on your environment. What matters is that one source dominates and that its query pattern does not match any legitimate workload behavior.

The two hostile explanations have distinct shapes:

• DGA malware generates pseudo-random domain names (high entropy, consonant-heavy, often long labels) against external zones, hoping one resolves to a live C2 server. Most fail, hence the NXDOMAIN flood. Queries are typically forwarded through the forward plugin to upstream resolvers.
• Domain enumeration probes structured, guessable names: common service names, namespace patterns, sequential IDs. Against cluster.local, this is an attacker mapping your internal service topology for lateral movement. Against external zones, it is subdomain brute-forcing.

Both leave the same fingerprint in aggregate metrics: elevated NXDOMAIN responses, a growing denial cache, and possibly a dropping cache hit ratio. The per-source attribution only exists in logs.

flowchart TD
  A[NXDOMAIN rate elevated or log alert fires] --> B{One source IP dominates?}
  B -- "No, spread across many pods" --> C[Likely ndots search expansion or app misconfig - not this runbook]
  B -- "Yes" --> D{What zone are the queries in?}
  D -- "cluster.local" --> E{Names structured or guessable?}
  E -- "Yes: common service names, patterns" --> F[Domain enumeration - reconnaissance]
  E -- "No: typo of one real name, tight loop" --> G[Broken app config or retry loop]
  D -- "External zone" --> H{Names high-entropy and random?}
  H -- "Yes" --> I[Probable DGA malware - treat pod as compromised]
  H -- "No: single external name repeated" --> G

Common causes

The last two rows are the benign lookalikes. Some legitimate applications generate moderate NXDOMAIN rates through DNS-based service-discovery probing, so absolute thresholds depend on the environment. The distinguishing question is always: does this client’s query pattern match what its workload is supposed to do?

Quick checks

These are read-only and safe to run during an incident.

# 1. Confirm the aggregate NXDOMAIN rate (all clients, per zone)
kubectl exec -n kube-system deploy/coredns -- \
  wget -qO- http://localhost:9153/metrics | grep 'coredns_dns_responses_total' | grep 'NXDOMAIN'

# 2. Check denial cache growth - recon and DGA inflate it
kubectl exec -n kube-system deploy/coredns -- \
  wget -qO- http://localhost:9153/metrics | grep '^coredns_cache_entries'

# 3. Aggregate NXDOMAINs by client IP from logs (requires the log plugin).
#    Field position for the client IP depends on your log format; verify against
#    one sample line before trusting the awk column.
kubectl logs -n kube-system deploy/coredns --since=1h | grep NXDOMAIN | \
  awk '{print $3}' | sort | uniq -c | sort -rn | head -20

# 4. Map the offending source IP to a pod
kubectl get pods --all-namespaces -o wide | grep <source-ip>

# 5. Sample the actual query names from that client (this is the decisive check)
kubectl logs -n kube-system deploy/coredns --since=10m | grep NXDOMAIN | \
  grep <source-ip> | head -50

If check 3 returns nothing useful, the log plugin is not enabled. See the diagnosis section; without query logs you are working blind on per-source attribution, because CoreDNS exposes no per-client-IP metric.

How to diagnose it

1. Confirm the anomaly is real and concentrated. Compare current NXDOMAIN rate against your rolling baseline using coredns_dns_responses_total{rcode="NXDOMAIN"}. Then check denial cache entries via coredns_cache_entries{type="denial"}. Growing denial entries with stable success entries is the classic fingerprint of scanning or DGA traffic. If total NXDOMAIN is flat and spread across many clients, you are likely looking at normal search-domain noise, not this problem.

2. Enable query logging if it is not already on. The log plugin is usually disabled in production because of volume. Enable it scoped to denial-class responses so you capture NXDOMAIN without logging every successful query: the log plugin supports class filtering, and class denial covers NXDOMAIN and NODATA responses. Edit the CoreDNS ConfigMap:

kubectl edit cm coredns -n kube-system

Add or adjust the log directive inside the relevant server block, then let the reload plugin pick it up (CoreDNS polls the Corefile roughly every 30 seconds) or roll the pods. Watch coredns_reload_failed_total after the change to confirm the new Corefile actually loaded. Note that even denial-class logging can be heavy on a large cluster; scope it to the server blocks you need and be ready to revert.

3. Attribute by source IP. Run quick check 3 and identify the dominant client. Map it to a pod with kubectl get pods -o wide. If the IP does not map to any current pod, it may be a host-level process, a pod that churned, or something reaching CoreDNS from outside the pod network. Check node conntrack entries and the kube-dns Service endpoints accordingly.

4. Classify the query names. This is the step that decides everything. Look at 50 to 100 sample names from the offending source:

   • Random, high-entropy external names, mostly unique, near-100% NXDOMAIN: probable DGA. Treat the pod as compromised.
   • Structured internal names probing cluster.local: enumeration. Also treat as hostile, but check whether the client is a security scanner your own team deployed before escalating.
   • One name or a handful of names in a tight loop: misconfiguration. Check whether the Service was renamed or deleted, or whether the app config points at the wrong namespace.
   • The same base name tried against every search suffix: ndots amplification, which is a performance problem, not a security one.

5. Corroborate with workload context. For the suspect pod: when was it deployed, what image does it run, what else is it doing? A DGA-infected pod usually shows other anomalies (egress connections to rare destinations, unexpected processes). A misconfigured pod shows nothing else suspicious. Check for companion signals: AXFR queries (coredns_dns_requests_total{type="AXFR"}) or ANY query spikes in the same timeframe indicate broader reconnaissance.

6. Decide and act. Compromised pod: isolate it (network policy, or cordon/quarantine per your incident response process) and preserve logs before deleting anything. Misconfigured pod: fix the name or the retry behavior. Scanner: get it allow-listed and rate-limited.

Metrics and signals to monitor

Do not alert on absolute NXDOMAIN count. It fluctuates with total traffic and is dominated by benign search-domain misses. Alert on the ratio change against baseline, and on the log-derived per-source concentration.

Fixes

Compromised pod (DGA)

There is no CoreDNS fix; the fix is incident response. Isolate the pod from the network first, then preserve evidence (CoreDNS logs covering the window, the pod spec, the image digest) before terminating it. Identify how the workload was compromised: vulnerable image, exposed credential, supply chain. Deleting the pod without isolating and capturing evidence destroys the trail and, if the cause is a bad image, the replacement pod gets reinfected.

Domain enumeration

Same isolation logic applies if the source is hostile. If the enumeration is coming from inside the cluster, treat the source pod as compromised until proven otherwise: a healthy workload has no reason to probe for which service names exist. If it turns out to be an internal security scanner, move it to an allow-listed identity and constrain it.

Misconfigured application

Fix the name, the namespace, or the retry policy at the source. A tight retry loop with no backoff against a nonexistent name is worth fixing even when benign: it pollutes the denial cache and adds load for nothing. The denial cache means CoreDNS itself absorbs repeated queries cheaply, but the client still pays latency per attempt.

Rate limiting at CoreDNS (optional, with caveats)

If you need to blunt a flood at the DNS layer itself, the external rrl plugin supports per-client response rate limiting, including an nxdomains-per-second allowance, and exports a per-client metric (coredns_rrl_responses_exceeded_total). Two caveats: it is not in the stock CoreDNS image, so using it means building and maintaining a custom image, which most managed-Kubernetes deployments do not do. And rate limiting treats the symptom, not the cause: a compromised pod that gets rate-limited at DNS is still compromised. Use it as a pressure valve during response, not as the resolution. The nxdomains-per-second option and the coredns_rrl_responses_exceeded_total metric are confirmed in the external rrl plugin’s documentation.

Prevention

• Keep scoped query logging on permanently. Logging only the denial class keeps volume manageable while preserving the attribution data you need. Discovering during an incident that logs are off is the common failure.
• Alert on NXDOMAIN ratio shifts and denial cache growth, never on absolute counts. Both are metric-level tripwires that fire before you ever open logs.
• Baseline per-workload DNS behavior. Knowing which namespaces legitimately probe for dependencies makes the anomaly obvious instead of ambiguous.
• Watch companion signals. AXFR and ANY query rates, response size distribution, and query type mix catch reconnaissance that does not manifest as NXDOMAIN.
• Constrain egress. Pods that cannot reach arbitrary external IPs limit what DGA malware can do even when a domain does resolve. Network policy is the containment layer DNS monitoring cannot be.

How Netdata helps

• Netdata collects the CoreDNS Prometheus endpoint, so the aggregate tripwires in this article (NXDOMAIN response rate per zone, denial cache entries, cache hit ratio, request type distribution) are charted per second and per pod, letting you spot divergence between replicas that aggregate views hide.
• Anomaly detection on the NXDOMAIN rate and denial cache growth surfaces the deviation-from-baseline pattern this symptom produces, without hand-tuned static thresholds against a noisy metric.
• Per-pod charts for coredns_dns_requests_total{type="AXFR"} and {type="ANY"} let you correlate the NXDOMAIN flood window with companion reconnaissance signals on the same timeline.
• Correlating CoreDNS metrics with node-level signals (conntrack utilization, UDP buffer errors) tells you whether a genuine flood is also stressing the node’s network layer, which changes the blast radius of your response.
• Because CoreDNS exposes no per-source-IP metric, Netdata’s role here is the aggregate early warning and correlation layer; per-client attribution remains log-based, as described above.

Related guides

• CoreDNS 5-second DNS timeout: the Kubernetes glibc A+AAAA conntrack race
• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS cache collapse: the cold-cache thundering herd after a rollout
• CoreDNS cache evictions: the cache is too small for the working set
• CoreDNS cache hit ratio dropping: latency and upstream load climbing together
• CoreDNS conntrack table full: silent UDP packet drops with a node-wide blast radius
• CoreDNS Corefile parse error: startup failures and invalid plugin config
• CoreDNS CPU throttling: CFS limits making a green dashboard lie about latency
• CoreDNS CrashLoopBackOff: triaging loop, OOM, config, and port-bind causes
• CoreDNS DNS programming latency: how long a Service takes to become resolvable
• CoreDNS not resolving external domains: the missing catch-all forward zone
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed