High-cardinality metrics at scale: why the standard playbook is wrong

The “high cardinality is expensive” sentence has become observability’s version of “in this economy” — said so often that nobody questions whether it’s true. Every vendor pricing page invokes it. Every glossary article repeats it. Every architecture diagram shows aggregation buffers placed before the storage layer.

But “high cardinality is expensive” is not a fact about the universe. It’s a fact about one architectural choice — centralizing time-series storage and querying it through an index that scales with unique series. Once you accept that choice, everything follows: you pay per metric, you drop labels you wish you’d kept, you pre-aggregate before storage, and you discover the bug you were debugging only existed at full resolution that you already threw away.

There’s another way. This post is about why it works, and what it costs to take it.

What “high cardinality” actually means

Cardinality, in this context, is the count of unique (metric_name, label_set) combinations a system tracks. A counter called http_requests_total has cardinality 1. The same counter with a path label has cardinality equal to the number of distinct paths. Add a user_id label and the cardinality explodes to number of paths × number of users seen. Add a request_id label and you’ve broken the cardinality budget of most TSDBs in production.

The reason cardinality matters for centralized TSDBs is specifically the inverted index. Systems like Prometheus, Mimir, Thanos, Datadog’s metric store, and OpenTSDB all maintain an index from (label_key, label_value) pairs back to the series they appear in. The index lets queries like http_requests_total{path="/checkout", region="eu"} resolve quickly. It also means every new label combination adds an index entry, and every active series costs memory.

When the vendor pricing page says “high cardinality is expensive,” that’s what they mean: their architecture grades active-series count or unique metric count or label combinations directly, because that’s what costs them money to operate. It’s not a falsehood. It’s a fact about that architecture.

The standard playbook

Walk into any “high-cardinality observability” page on a vendor site, glossary entry, or AIOps consultant’s slide deck, and you’ll find the same three pieces of advice:

1. Aggregate before storage. Push the data through a streaming buffer (Cribl, Vector, OTel Collector, Fluent Bit) and drop or roll up labels before the TSDB sees them.
2. Pay for cardinality. Use the vendor’s premium tier that promises “unlimited” cardinality, knowing the bill scales with it anyway.
3. Drop labels you don’t need. Specifically, drop high-cardinality labels — request_id, user_id, session_id — and accept that you cannot ask post-hoc questions about specific users, sessions, or requests.

This playbook is internally consistent and works. It also costs you the most valuable observability signal you have: full-resolution data captured at the moment the system was misbehaving. You can’t get it back. The request_id you dropped to save cardinality was the one a customer just escalated.

The architectural alternative

The alternative is: don’t centralize. Keep the time-series data on the host that produced it, at full resolution, and federate queries across hosts only when a question demands cross-host correlation.

This is the architecture Netdata uses. Each Netdata Agent runs an embedded time-series database that holds the full-resolution per-second metrics for that node. There is no central index across all labels of all metrics of all hosts. Cardinality on each node is bounded by what that node generates — which is high, but per-node bounded, not multiplied across the fleet.

Some implications fall out immediately:

• Cardinality is sharded by host. If you have 100 hosts, you have 100 small TSDBs, each holding the full-resolution series produced by its host. The fleet-wide cardinality is the sum, not the cross-product.
• There is no global index to manage. Queries that need cross-host aggregation are computed by federating per-host results, not by reading a single global index.
• The pricing dynamic changes. If your bill grows with hosts (not with active series), label-richness doesn’t compound your cost. Adding a user_id label to a metric is free at the storage layer.

The trade-off is honest: cross-host exploratory querying (“show me every request with error=true across the entire fleet for the last hour”) is slower than asking the same question of a single centralized index. The architecture optimizes for per-host detail at full resolution, not for fleet-wide ad-hoc OLAP queries on metrics.

For infrastructure observability, the trade is usually right. Most incidents are localized to a host, a pod, a service, a region. The post-mortem question is “what was happening on this specific node at this specific moment” — not “give me a SQL query across 50 million series.”

What “ML on every metric” looks like once cardinality isn’t a tax

Here’s a corollary most cardinality discussions miss. If your architecture bills you per metric or per active series, you can’t afford to run anomaly detection on every metric. You curate a subset of “golden signals” — usually the famous four: latency, traffic, errors, saturation — and apply ML there. The other 95% of your metrics get static-threshold alerting or no alerting at all.

If your architecture doesn’t bill you per metric, you can score everything. Netdata trains 18 unsupervised models per metric, on each agent, on rolling windows of recent data. Every collected metric gets ML treatment — not because the marketing brief said so, but because the per-metric scoring cost is computed locally on the same host that captured the metric, on hardware you’re already paying for to run the workload.

The downstream effect is the one buyers actually care about: when 18 independent models disagree about whether the same observation is anomalous, you don’t fire an alert. When they all agree, you do. The consensus-vote architecture isn’t enabled by ML cleverness — it’s enabled by the cost structure that lets you afford ML on every metric in the first place.

What this doesn’t solve

Honesty section.

This architecture is not a free win on every dimension:

• Long-term retention and cross-fleet OLAP. If your workflow is “give me a year of every metric across every host with arbitrary label combinations as the query plan demands,” a centralized columnar store (VictoriaMetrics, Mimir, ClickHouse) will outrun federated per-host queries. Many teams pair the architectures: full-resolution per-host for real-time, downsampled centralized for long-term.
• Distributed tracing. Tracing fundamentally requires per-request correlation across services, which is a centralized data problem. Netdata’s tracing roadmap is open; for now, pair with Jaeger or Tempo if traces drive your workflow.
• Cross-host exploratory analytics on metrics. If you regularly run “show me every error metric across every region grouped by customer-tier” — that’s a centralized-OLAP question, and a host-sharded architecture is the wrong answer for it.

The recurring pattern across these “doesn’t solve” cases is the same: when the workload is genuinely cross-host exploratory, centralized stores win. When the workload is per-host operational, federated full-resolution wins. Most infrastructure observability is the second category. Most marketing assumes the first.

What to do tomorrow

A small, concrete set of moves any team can make this week:

1. Audit which labels you’ve been told to drop. For each one, ask “if I had it captured at full resolution on the host, what question could I now answer?” The answers are usually the ones that matter most in incident review.
2. Look at your monitoring bill’s cardinality dimension. Whether it’s Datadog’s custom metrics line, Grafana Cloud’s active series, or Splunk’s ingest meter — what percentage of the bill is cardinality-driven? That number tells you how much of your budget is going to the standard playbook.
3. Try an architectural counterfactual. Run Netdata on one host in production for a week, alongside whatever you’re using now. Don’t migrate — just collect both. Compare the resolution. The per-second resolution shows you transients the centralized minute-aggregated metric will literally never see.

The standard cardinality playbook is the right answer to a question — “how do I run a centralized TSDB at scale” — that isn’t actually the question. The question is: what infrastructure observability do I want, and what architecture should I pick to make it economically possible? Once you ask it that way, paying for cardinality stops being a law of nature and starts being a choice.

Netdata’s full-resolution, per-second, per-host architecture is open source (AGPL) and packaged as a single binary you can install on any node with one command. The optional managed Cloud is per-node pricing — there is no cardinality dimension in the bill. The team is happy to talk through how this works on your specific workload; book a free demo or browse the features documentation for the architecture details.