CoreDNS response size distribution: large answers, TCP pressure, and amplification

Most CoreDNS dashboards track QPS, latency, and SERVFAIL. Response size rarely makes the cut until something odd happens: TCP/53 traffic climbs for no obvious reason, a workload starts resolving incomplete answers, or a security review asks whether your resolvers could be used as DDoS reflectors. All three questions are answered by the same signal: the distribution of DNS response sizes CoreDNS is sending.

The metric is coredns_dns_response_size_bytes, a histogram exported by the prometheus plugin with labels server, zone, view, and proto (udp/tcp). This article covers what a healthy distribution looks like, what pushes it toward large answers, why large answers create TCP pressure, and how to separate benign large record sets from amplification abuse using the labels you already have. It is a reference for interpreting the signal; the related guides at the end cover the concrete failure modes.

What the metric is and why it matters

coredns_dns_response_size_bytes records the wire size of every response CoreDNS sends, bucketed as a Prometheus histogram. Scrape it directly with:

# Inspect the response size histogram
curl -s http://localhost:9153/metrics | grep 'coredns_dns_response_size_bytes'

Two labels do most of the diagnostic work:

• proto splits observations into udp and tcp. Because clients only fall back to TCP after receiving a truncated UDP answer, the TCP share of this histogram is a proxy for truncation pressure. There is no direct Prometheus metric for the TC (truncated) bit, so this split is the closest observable signal short of packet capture or dnstap.
• zone tells you which part of your namespace is producing large answers. Large responses in cluster.local point at in-cluster record sets (headless Services with many endpoints, big SRV records). Large responses in forwarded zones point at upstream content (large TXT records, DNSSEC-signed answers).

A sustained shift in this distribution matters for two distinct reasons. Operationally, large UDP responses get truncated, clients retry over TCP, and TCP/53 has very different cost and failure characteristics than UDP/53. From a security perspective, a resolver answering with responses many times larger than the requests is exactly what a DNS amplification reflector looks like.

How large answers turn into TCP pressure

The mechanism is fixed by the DNS protocol and CoreDNS’s buffer size policy:

1. A client sends a UDP query advertising an EDNS0 buffer size. Non-EDNS0 clients implicitly advertise 512 bytes.
2. If the answer fits the advertised buffer, CoreDNS sends it over UDP. If it does not, CoreDNS sends a truncated response with the TC bit set.
3. A well-behaved client sees TC=1 and retries the same query over TCP.
4. CoreDNS answers the TCP query with the full response. The observation lands in the proto="tcp" bucket of the histogram.

CoreDNS’s bufsize plugin caps the client-advertised EDNS0 buffer size. The default is 1232 bytes, chosen to avoid IP fragmentation in line with DNS Flag Day 2020. Two version-dependent facts matter here:

• Earlier CoreDNS releases defaulted bufsize to 512, and the default was raised to 1232 in CoreDNS 1.11.0. If you upgrade across that boundary, the UDP ceiling rises, which reduces truncation and TCP fallback for any zone that was serving answers between 512 and 1232 bytes over UDP.
• The bufsize plugin can only reduce the advertised size, not raise it, and its valid range is 512-4096. There is no separate server-side “max UDP response size” knob: the client’s advertised buffer, capped by bufsize, controls the ceiling.

flowchart TD
  Q[Client UDP query with EDNS0 bufsize] --> F{Answer fits buffer?}
  F -->|yes| U[Full answer over UDP - proto=udp]
  F -->|no| T[Truncated answer TC=1 over UDP]
  T --> R{Client retries over TCP?}
  R -->|yes| A[Full answer over TCP - proto=tcp]
  R -->|no, firewall blocks TCP/53| X[Client fails or stalls silently]
  R -->|TC bit lost upstream| X

Two failure edges in that flow are worth knowing:

• TCP/53 blocked. If a firewall or NetworkPolicy blocks TCP/53 to CoreDNS, the retry never arrives. The client got a truncated answer it cannot use and cannot complete. This fails silently from CoreDNS’s perspective: the histogram shows the truncated UDP answer, but there is no corresponding TCP observation, and no error metric increments.
• Lost TC bit. CoreDNS issue #4183: with neither force_tcp nor prefer_udp set, the forward plugin passed a truncated upstream response back to the client without the TC flag, delivering an incomplete answer the client never knows to retry. Fixed in CoreDNS 1.8.0, which retains the TC bit on truncated messages; CoreDNS 1.11.3 additionally made the forward plugin set the TC bit itself when an upstream response overflows the client’s buffer. If you are on an older build and see clients acting on partial data for names with many addresses, this is the mechanism to check.

The practical consequence: a response size distribution shifting right is not just a histogram curiosity. It converts cheap, connectionless UDP queries into TCP sessions with handshake, state, and file descriptor cost, and it introduces two silent failure modes that no error counter will surface.

What a healthy distribution looks like

Baselines are deployment-specific, but some anchors hold broadly:

• Most answers in a Kubernetes cluster are small: A/AAAA records for Services, NXDOMAIN responses from search-domain expansion. These sit well under 512 bytes.
• As rules of thumb, not vendor thresholds: responses above 4096 bytes are anomalous for most deployments, and a sustained shift greater than 2x in average response size over 10 minutes warrants investigation.
• The proto split should be overwhelmingly UDP. TCP exists for truncation fallback and legitimately large answers, but a TCP share that climbs without a corresponding change in record sets is a truncation-pressure signal.

Watch the distribution, not just the mean. A small number of very large answers moves the mean without telling you whether you have one giant TXT record or a systemic truncation problem. The histogram buckets, sliced by zone and proto, answer that.

The histogram’s bucket boundaries in current CoreDNS are 0, 100, 200, 300, 400, 511, 1023, 2047, 4095, 8291, 16000, 32000, 48000, and 64000 bytes. They are not documented in the metrics README, so verify them against the _bucket series from a live scrape before building alerts on specific byte thresholds, since they can change between releases.

Benign causes of large answers

Before treating a rightward shift as abuse, rule out the legitimate drivers:

Large in-cluster record sets. Headless Services with many endpoints return one A record per endpoint in a single answer. At a few dozen endpoints, answers exceed 1232 bytes and every lookup truncates and retries over TCP. SRV records for StatefulSets and gRPC service discovery have the same shape. This shows up as large responses in the cluster.local zone, correlated with endpoint count growth.

Large upstream content. Big TXT records (SPF policies with many includes, domain verification chains) and DNSSEC-signed answers with attached signatures routinely exceed 1232 bytes. This shows up in forwarded zones.

A bufsize change. If you upgraded CoreDNS across the bufsize default change or edited the bufsize directive, the truncation threshold moved. The record sets did not change; the ceiling did. Expect a step change in the proto="tcp" share at the moment of the change, then a new steady state.

Non-EDNS0 clients. Legacy clients advertising no EDNS0 buffer are held to 512 bytes. Even modestly sized answers truncate for them, inflating the TCP share out of proportion to actual answer sizes.

Amplification and abuse signals

The security reading of the same histogram: DNS amplification works because a small spoofed query produces a large answer delivered to a victim. CoreDNS used as a reflector shows up as a sudden, sustained shift toward large responses without any matching change in legitimate record sets.

Two correlations, using metrics you already collect, separate abuse from benign bigness:

Correlate with ANY query rate. ANY queries return all records for a name and produce the largest responses per query, which is why attackers favor them. Check coredns_dns_requests_total{type="ANY"}:

# Check ANY query volume against total traffic
curl -s http://localhost:9153/metrics | grep 'coredns_dns_requests_total' | grep 'type="ANY"'

A response size shift accompanied by an ANY spike is the amplification signature. As a heuristic, ANY queries above 5% of total traffic are unusual, and a spike from a zero baseline is a strong indicator. A response size shift with a flat, normal ANY rate points at benign large record sets instead.

Correlate with the proto split. An amplification reflector sends large answers over UDP (the attacker does not complete a TCP handshake with a spoofed source). Benign truncation pressure shows the opposite shape: large answers landing in proto="tcp" buckets because real clients completed the retry. Large UDP responses with no matching TCP retry activity are the pattern to investigate.

Also check AXFR (type="AXFR") while you are in the request-type distribution: zone transfer attempts are reconnaissance, not amplification, but they belong in the same review.

The honest limits of what metrics alone can tell you:

• CoreDNS exposes no per-source-IP metrics, so you cannot attribute large responses to specific clients from Prometheus alone. That requires the log plugin or dnstap.
• There is no truncation counter. The proto split is an inference, not a measurement.
• coredns_dns_request_size_bytes exists as a companion histogram of query wire size, so the response-to-request size ratio is conceptually the amplification factor, though CoreDNS documents no formula for deriving it from the two histograms.

Signals to watch in production

Alert on the distribution with ratios and trends: average response size versus a rolling baseline, and TCP share of responses versus its baseline. Absolute byte thresholds, other than the >4096 anomaly marker, will false-positive whenever someone legitimately adds endpoints to a headless Service.

Operational responses to a confirmed shift

What you do depends on which cause the correlation pointed at. These are adjustments, not emergency fixes.

• Benign large in-cluster records: Verify TCP/53 is actually reachable to CoreDNS (firewalls and NetworkPolicies included), because your clients now depend on it. Confirm you are on a CoreDNS release where the forward plugin sets the TC bit on truncated responses so clients retry correctly. Then watch TCP share and FD usage as the endpoint count grows; this is a capacity trend, not an incident.
• Unexpected truncation after an upgrade: If the shift aligns with an upgrade that changed the bufsize default to 1232, decide deliberately whether to keep the fragmentation-safe default or set bufsize higher in the Corefile for zones with legitimately large answers, accepting the fragmentation risk the default exists to avoid.
• Suspected amplification: Validate the ANY correlation, then move to log or dnstap data to identify sources, since metrics cannot attribute per-client. If CoreDNS is reachable from untrusted networks and answering with large responses, that is an exposure problem to fix at the network edge, not a Corefile tuning problem.

How Netdata helps

• Netdata charts coredns_dns_response_size_bytes per second, so the rightward shift in the distribution is visible as it develops rather than after a 10-minute alert window closes.
• The proto label breakdown renders the UDP/TCP split side by side, which is the fastest way to see truncation pressure building before clients start reporting failures.
• Request-type distribution charts make the ANY-query correlation a glance instead of a PromQL query: response size spike plus ANY spike is the amplification pattern; response size spike alone points at record sets.
• Correlating response size with request latency and FD usage on the same dashboard connects the mechanism: large answers, TCP fallback, connection cost, tail latency.
• Anomaly detection on the size histogram flags distribution shifts that sit below absolute thresholds but well outside the learned baseline.

Related guides

• CoreDNS 5-second DNS timeout: the Kubernetes glibc A+AAAA conntrack race
• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS cache collapse: the cold-cache thundering herd after a rollout
• CoreDNS cache evictions: the cache is too small for the working set
• CoreDNS cache hit ratio dropping: latency and upstream load climbing together
• CoreDNS conntrack table full: silent UDP packet drops with a node-wide blast radius
• CoreDNS Corefile parse error: startup failures and invalid plugin config
• CoreDNS CPU throttling: CFS limits making a green dashboard lie about latency
• CoreDNS CrashLoopBackOff: triaging loop, OOM, config, and port-bind causes
• CoreDNS DNS programming latency: how long a Service takes to become resolvable
• CoreDNS not resolving external domains: the missing catch-all forward zone
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed