Envoy stats cardinality explosion: the stats region fills and new metrics vanish

Envoy builds each stat name from the resource names in your config: clusters, listeners, HTTP connection managers, routes. When those names are stable, cardinality is bounded. When they are dynamic, every distinct value mints a new metric. The budget that backs those stats fills, and new stats stop appearing with no error and no log line.

The failure is quiet by design. Envoy treats stat registration as best-effort relative to its memory budget. A missing metric raises no alert, increments no hot-path error counter, and never appears in access logs. You usually discover it weeks later, when an incident sends you hunting for a per-cluster or per-route metric that was never recorded.

The framing most operators inherit, “the stats region fills”, describes the pre-1.11.0 model where stats lived in a fixed-size shared-memory region used for hot restart. That backend is gone. Since 1.11.0, stats are heap-allocated and the old fixed region does not exist. The silent-disappearance symptom still exists, but for different reasons and under different signals. This article covers both, because the diagnostics and the mitigations overlap.

What this means

Two distinct mechanisms produce “new metrics vanish”, and you need to know which one you are in.

Classic shared-memory region (Envoy before 1.11.0): stats lived in a fixed-size shared-memory block shared between hot-restart processes. When the block was full, Envoy stopped registering new stats. There was no error. The tell-tale sign was a stat count that climbed steadily and then went flat. The default region was roughly 16 MB, with each stat consuming on the order of 100 to 200 bytes for name, value, and lock (operator estimate; the exact region size was never documented). Envoy 1.11.0 removed this backend: its changelog notes stats are “no longer shared between hot restart parent/child via shared memory, but rather by RPC”.

Current Envoy (1.11.0 and later): stats are heap-allocated and backed by a SymbolTable that interns dot-delimited name tokens. There is no fixed slot count. The constraints are memory and, where you configure it, per-sink label-cardinality caps. The relevant failure modes are:

• Heap pressure from the sheer number of stat objects and interned symbols, which eventually trips the overload manager or the container OOM killer.
• Per-sink label-cardinality caps, tracked by server.stats_overflow. This counter increments when a stat lookup or creation is dropped because a configured cardinality limit was reached. It tracks per-sink label caps, not the global stat count.

server.stats_overflow exists in current releases as three per-type counters — stats_overflow.counter, stats_overflow.gauge, and stats_overflow.histogram — each counting lookup-or-creation attempts dropped when the configured limit is reached. No still-supported release line ships the shared-memory stats backend; it was removed in 1.11.0.

In both cases the operator-visible symptom is identical: a metric you expected to see is absent, and nothing in Envoy told you it was dropped. The first diagnostic question is therefore not “why is this metric missing” but “how many stats do I actually have, and is that number still growing”.

flowchart TD
    A["Dynamic stat names
cluster / route / tag values"] --> B["Unique metric count climbs"]
    B --> C["Region or heap budget exhausted"]
    C --> D["New stats silently fail to register"]
    D --> E["Stat count plateaus
no error, no log"]
    E --> F["Monitoring blind spots
grow over time"]

Common causes

Quick checks

All commands are read-only except where noted. Set ADMIN_PORT to your admin port: 9901 by default, 15000 in Istio sidecar mode.

# Total stat names exposed. Run twice, minutes apart, to see whether it is still growing.
curl -s http://localhost:${ADMIN_PORT:-9901}/stats | wc -l

# Only stats that have actually been touched. A tighter signal of live cardinality.
curl -s "http://localhost:${ADMIN_PORT:-9901}/stats?usedonly" | wc -l

# Overflow counters that track dropped lookups under per-sink cardinality caps (current Envoy).
curl -s http://localhost:${ADMIN_PORT:-9901}/stats | grep 'server.stats_overflow'

# Start collecting recent symbol-table lookups, then read the 20 most recent.
# This is a write to admin state, but harmless: it enables a ring buffer of recent lookups.
curl -s -X POST http://localhost:${ADMIN_PORT:-9901}/stats/recentlookups/enable
curl -s http://localhost:${ADMIN_PORT:-9901}/stats/recentlookups
curl -s -X POST http://localhost:${ADMIN_PORT:-9901}/stats/recentlookups/disable

# Memory breakdown to see whether stats are driving heap growth.
curl -s http://localhost:${ADMIN_PORT:-9901}/memory

# Memory gauges from the stats endpoint.
curl -s http://localhost:${ADMIN_PORT:-9901}/stats | grep 'server.memory'

A stat count that climbs sharply and then goes flat is the signature. In the classic model, the plateau means the region is full. In current Envoy, a plateau combined with rising server.stats_overflow or rising server.memory_allocated points to the same class of problem expressed differently.

How to diagnose it

1. Confirm the plateau. Sample curl /stats | wc -l two or three times over a few minutes. A number that never moves during a period when traffic and config are changing is suspicious. Cross-check with ?usedonly to separate registered-but-unused stats from genuinely live cardinality.
2. Find the offending pattern. Enable /stats/recentlookups and drive a representative request. The 20 most recent lookups usually reveal the culprit: a cluster name with an embedded FQDN, a tag carrying a user or request identifier, or a route name derived from a dynamic value.
3. Correlate with config churn. Compare stat-count jumps with cluster.<name>.update_success bursts or cluster_manager.cluster_added increments. If cardinality spikes track xDS pushes, the source is dynamic resources, not steady-state traffic.
4. Separate registration loss from scrape loss. If a metric is missing from Prometheus but present in curl /stats, the problem is in the scraper (interval, relabeling, dropped series), not in Envoy. If it is absent from /stats entirely, Envoy never registered it.
5. Check the overflow and memory signals together. Nonzero server.stats_overflow confirms per-sink drops. Rising server.memory_allocated without a matching traffic increase confirms stats are consuming real heap. Either confirms the cardinality problem is live, not theoretical.

Metrics and signals to monitor

There is no dedicated “total registered stats” gauge in current Envoy; counting the entries returned by the /stats endpoint (or the length of a Prometheus scrape) remains the way to measure it.

Fixes

Apply a stats_matcher

The primary lever is stats_matcher in stats_config. It has three mutually exclusive modes: reject_all (record nothing), exclusion_list (record everything except matches), and inclusion_list (record nothing except matches). For an uncontrolled explosion, an exclusion_list keyed on the offending pattern is the fastest containment. For a sidecar where you want a small, known set of stats, an inclusion_list is safer because it bounds cardinality by construction.

# Bootstrap-level sketch. Verify the StringMatcher oneof shape for your Envoy version
# against config.metrics.v3.StatsMatcher.
stats_config:
  stats_matcher:
    exclusion_list:
      - safe_regex:
          regex: ".*(user_id|request_id|trace_id).*"

Two cautions. First, the Envoy docs warn that excluding stats can affect Envoy behavior in undocumented ways, so exclude narrowly, not broadly. Second, an inclusion_list that is too aggressive will silently hide stats you later need during an incident. Treat the matcher as code: review it, version it, and diff stat count before and after every change.

Remove dynamic values from stat names

The durable fix is to stop generating dynamic stat names. In service mesh, the classic offender is the full cluster FQDN embedded in the stat path, for example the outbound|8080||fortio-server-l2.mark.svc.cluster.local pattern. Istio relies on statsConfig.statsTags regexes to capture and strip those tokens down to a stable label set. If those regexes do not match, the full name lands in the Prometheus metric name and cardinality tracks your endpoint count. Fix the tag-extraction regexes, not the symptom.

Reduce per-route and per-cluster verbosity

If you enabled per-route or per-cluster dynamic stats to chase a previous incident, scope them to the routes and clusters that actually need them. Broad per-route instrumentation across thousands of dynamic routes is the most common path from “we added observability” to “we lost observability”.

Tune Istio proxyStatsMatcher carefully

Istio configures Envoy to record a minimal stat set by default. Adding stats through proxyStatsMatcher multiplies across every sidecar and every series the scraper retains. Add stats one family at a time and measure the Prometheus series count delta after each addition.

Raise the budget only as a stopgap

In the classic model, the lever was the shared-memory region size. In current Envoy the lever is container memory and, where relevant, per-sink cardinality caps. Raising memory buys time but does not fix unbounded growth. If you raise the budget, set a stat-count alarm at the new ceiling so the next plateau is loud instead of silent.

Prevention

• Track stat count as a first-class metric. Export curl /stats | wc -l and curl /stats?usedonly | wc -l on a schedule. A plateau after growth is the single most reliable signal that registration has stopped.
• Alarm on the overflow counter. Any nonzero server.stats_overflow means stats are already being dropped. Treat it as an incident, not baseline.
• Avoid dynamic stat names by design. Cluster, route, and virtual host names are stat-name components. If a name is generated from request data, it does not belong in a stat.
• Gate config changes that add cardinality. Diff stat count before and after any xDS change, any proxyStatsMatcher addition, and any new dynamic cluster source.
• Use efficient scraping. Prefer /stats/prometheus over text /stats, use a scrape interval of 30 seconds or more on high-traffic proxies, and consider ?usedonly to cut scraper load and expose live cardinality.
• Review stats_matcher periodically. Matchers accumulate. An exclusion added during an incident can mask a new stat family you need later.

How Netdata helps

• Stat-count trend visibility. When you export curl /stats | wc -l as a metric, Netdata’s per-second resolution makes a plateau or a growth spike obvious against the normal baseline, instead of discovering it weeks later.
• Memory correlation. server.memory_allocated, server.memory_heap_size, and server.memory_physical_size plotted alongside stat count show whether cardinality is the driver of heap growth or a symptom of something else.
• Overflow counter alerting. Nonzero server.stats_overflow is a clean anomaly signal. Anomaly detection on that counter catches the first drop, not the thousandth.
• Config-churn correlation. xDS signals such as update_success, update_rejected, and cluster_manager.cluster_added let you line up a cardinality jump with the deployment that caused it, which shortens root cause from “something changed” to “this specific push”.
• Overload-manager context. When heap pressure from stats tips the overload manager into stop_accepting_connections, correlating that action with the stat-count and memory trend tells you whether you have a cardinality problem or a traffic problem.

Related guides

• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy control_plane.connected_state = 0: running on stale xDS config
• Envoy membership_healthy dropping: reading the single most important cluster signal
• Envoy listener_create_failure: a listener config Envoy could not apply
• How Envoy actually works in production: a mental model for operators