$ guides / traefik / traefik-routing-table-explosion ▌

Operations Guides

Traefik routing table explosion: thousands of routes and slow rebuilds

Your Traefik instance is still passing traffic, but something is off. Memory climbs week over week. Every Ingress change causes a CPU blip and a small latency spike. Restart counts are creeping up, and the last restart was an OOM kill. When you look at the provider, you find thousands of Ingress or IngressRoute objects that nobody ever pruned.

This is the routing table explosion failure mode. Traefik rebuilds its entire routing table on every configuration change, holds the whole thing in memory, and pays a per-request matching cost for every router it knows about. The failure is slow and linear right up until the OOM killer makes it instant.

The trap is that nothing looks broken along the way. /ping returns 200. Error rates are normal. Latency is fine except for brief spikes that nobody can explain. The only honest early indicators are memory trending up, reload count growing, and the raw count of routers, services, and middlewares in the loaded configuration.

What this means

Traefik does not do incremental configuration updates. Every change from any provider triggers a full rebuild of the routing table. The rebuild cost is proportional to routers x middlewares x services, and it competes directly with request-serving work. With more than about 5,000 routes, a single rebuild can consume 50-200ms of CPU and briefly stall request handling while the handler swap completes.

Memory scales with the routing table as well. Each router, middleware chain, and service definition is a live object graph, and Traefik v2 and later use significantly more memory per route than v1 did due to the middleware chain architecture. Routing tables above 10,000 routes can consume hundreds of MB at rest, before you count connections, buffers, and metric cardinality.

On top of that, every router adds matching cost to every request. A request arriving at an entrypoint is evaluated against the router table in priority order, so table size is not just a memory problem, it is a steady-state CPU tax on all traffic.

flowchart TD
  A[Route count grows
Ingress / IngressRoute sprawl] --> B[Memory grows linearly
router + middleware + service objects]
  A --> C[Every change triggers
full routing table rebuild]
  C --> D[Rebuild CPU cost grows
routers x middlewares x services]
  D --> E[Latency spikes + CPU steal
during each reload]
  B --> F[RSS approaches container limit]
  F --> G[OOM kill
all connections dropped]
  E --> H[GC pressure rises
go_gc_duration_seconds climbs]
  H --> F

Common causes

Cause	What it looks like	First thing to check
Unbounded Ingress/IngressRoute growth	Router count climbs steadily; nobody deletes old objects	Count objects in the provider and in Traefik’s API
High provider churn on a large table	Frequent reloads, each one expensive; CPU spikes correlate with reload events	`traefik_config_reloads_total` rate vs router count
Regex-heavy router rules at scale	Heap dominated by compiled rule matching; higher memory per route than expected	Rule complexity in loaded routers; version-specific regexp memory behavior in v2
Per-router middleware duplication	Middleware count grows faster than router count, multiplying rebuild cost	Middleware count relative to router count
Per-route Prometheus labels	Metric cardinality grows with route count, inflating memory	Whether `addRoutersLabels` is enabled
Long-lived connections pinning old configs	Memory ratchets up after each reload and never returns	RSS after reloads with WebSocket/gRPC traffic present

Quick checks

All of these are read-only. The API endpoints assume the Traefik API/dashboard entrypoint is enabled and reachable on port 8080; adjust for your deployment.

# Count loaded routers, services, and middlewares (the core growth signal)
curl -s http://localhost:8080/api/http/routers | jq 'length'
curl -s http://localhost:8080/api/http/services | jq 'length'
curl -s http://localhost:8080/api/http/middlewares | jq 'length'

# Reload activity: is the table being rebuilt often?
curl -s http://localhost:8080/metrics | grep traefik_config_reloads_total

# Memory: resident set and Go heap
curl -s http://localhost:8080/metrics | grep -E 'process_resident_memory_bytes|go_memstats_heap_inuse_bytes'

# GC pressure
curl -s http://localhost:8080/metrics | grep go_gc_duration_seconds

# Goroutines (connection handling plus watchers)
curl -s http://localhost:8080/metrics | grep go_goroutines

# CPU consumed by the process (rate this over time)
curl -s http://localhost:8080/metrics | grep process_cpu_seconds_total

# Container limit context: what is Traefik actually using right now?
grep VmRSS /proc/$(pgrep traefik)/status

Two things to note. First, there is no Prometheus metric for router count, so the /api/http/* count checks are the direct measurement. Second, in Kubernetes, compare the counts against the cluster truth:

# What the provider thinks exists
kubectl get ingresses -A --no-headers | wc -l
kubectl get ingressroutes -A --no-headers | wc -l

If the provider count and the loaded count diverge significantly, you may also have a config freshness problem on top of the size problem.

How to diagnose it

Establish the object counts. Record routers, services, and middlewares from the API. This is your baseline. Low thousands is fine on healthy hardware; tens of thousands is the danger zone, where memory is measured in hundreds of MB just for the table.
Confirm the memory trend is config-driven. Plot process_resident_memory_bytes and go_memstats_heap_inuse_bytes over days. If heap grows in step with router count (or with reload events) rather than with traffic, the routing table is the driver, not a connection or goroutine leak. A goroutine leak shows go_goroutines climbing with heap; a routing table problem shows heap climbing with stable goroutines.
Measure the reload storm. Rate traefik_config_reloads_total over 5-10 minutes. In a busy cluster, pod churn alone can trigger near-continuous reloads. Now correlate: does process_cpu_seconds_total spike at each reload? Does traefik_entrypoint_request_duration_seconds show p99 jitter aligned with reload events? If yes, rebuild cost is already user-visible.
Check whether memory comes back after reloads. Watch RSS across several reloads. If each reload leaves RSS slightly higher and it never returns, suspect long-lived connections (WebSockets, gRPC) holding references to old configuration generations, or version-specific leaks. Multiple upstream reports describe memory that only resets on process restart; treat restart as a diagnostic, not a fix.
Check your version against known memory regressions. The v2 to v3 migration had a documented memory regression tied to the compress middleware, fixed in v3.3. Older v2 releases had a regexp compiler memory issue where Host() rules dominated the heap at high route counts. If you are on an affected version and see heap growth out of proportion to route count, upgrading may matter more than any config change.
Decide which failure you are closest to. If RSS is within 20% of the container limit, the OOM kill is your incident-in-waiting and capacity work is urgent. If memory is fine but rebuilds are visibly stalling requests, the rebuild cost is your incident and the fix is reducing table size or churn.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Router/service/middleware count (via API)	The primary growth signal; drives both memory and rebuild cost	Steady week-over-week growth; counts in the high thousands
`process_resident_memory_bytes`	Direct OOM runway indicator	> 80% of container limit sustained
`go_memstats_heap_inuse_bytes`	Live heap; separates table growth from transient buffers	Growing without corresponding traffic growth
`traefik_config_reloads_total` (rate)	How often the full rebuild runs	Rate climbing in step with provider churn
`process_cpu_seconds_total` (rate)	Rebuild cost and per-request matching cost	CPU spikes aligned with reload events
`go_gc_duration_seconds`	GC struggling with allocation churn from rebuilds	p99 pauses climbing
`go_goroutines`	Distinguishes table growth from connection leaks	Growth disconnected from traffic means leak, not table size
`traefik_entrypoint_request_duration_seconds` p99	Rebuild stalls surface here first	Intermittent spikes correlated with reloads, not sustained latency

Fixes

Reduce the route count

The only fix that addresses both memory and rebuild cost. Audit Ingress and IngressRoute objects for dead services, abandoned hostnames, per-customer or per-environment sprawl that should be consolidated, and duplicate routers covering the same hosts. In Kubernetes, ownership of Ingress objects is often diffuse; expect a report and a cleanup campaign rather than a quick delete.

Tradeoff: route consolidation (shared routers with broader rules, shared middleware chains) reduces object count but increases blast radius per change. A bad edit to a shared router affects everything behind it.

Throttle reload frequency

providers.providersThrottleDuration controls how long Traefik waits before accepting new refresh events after a reload, coalescing bursts of provider events into one rebuild. The default is 2s. In a high-churn environment, raising it to 5-10s trades config freshness for a large reduction in rebuild CPU.

Tradeoff: new services take longer to become routable and removed backends take longer to drain. Do not raise this if your deployment pipeline depends on sub-second route convergence.

Right-size memory limits

If cleanup will take time, raise the container memory limit so you are not one traffic spike away from OOM. A workable rule of thumb: the limit should be about 2x the observed stable peak heap, and steady state should stay below 70% of the limit at peak router count. This buys runway; it does not fix growth.

Split the configuration across instances

For genuinely large configurations, partition routes across multiple Traefik deployments: by namespace, by domain suffix, by team, or by tenant. Each instance keeps a small table, cheap rebuilds, and an independent failure domain. In Kubernetes this maps naturally to multiple IngressClasses.

Tradeoff: operational complexity. You now have N configs, N sets of metrics, and per-instance config freshness to watch. ACME and wildcard cert handling need a clear owner per partition.

Upgrade if you are on an affected version

If heap profiling or version history points at a known regression (compress middleware memory in early v3, regexp compiler memory in v2), upgrading is the highest-leverage fix. Confirm the fix version for your specific symptom before scheduling it.

Prevention

Track object counts as a first-class signal. Scrape or cron the /api/http/routers, /api/http/services, and /api/http/middlewares counts into your monitoring system. Growth rate is more useful than the absolute number; alert on slope, not threshold.
Set a route budget. Decide the maximum table size your memory limit and rebuild tolerance support, publish it, and review quarterly. Keep steady state below 70% of allocated memory at peak router count.
Gate route creation. If teams self-serve Ingress objects, add admission checks or CI linting that flags duplicate hosts, dead backends, and missing ownership labels.
Baseline reload cost. Know your reload rate and per-rebuild CPU cost at current table size. When either doubles, investigate before it becomes the incident.
Plan the partition early. Splitting a 20,000-route monolith under incident pressure is miserable. If growth trends say you will cross the comfort line in two quarters, start the multi-instance design now.

How Netdata helps

Netdata charts process_resident_memory_bytes and Go heap metrics per second, so the slow linear climb of a growing routing table is visible weeks before the OOM kill, not after.
Correlating traefik_config_reloads_total with CPU and entrypoint latency on one dashboard makes the rebuild-stall pattern obvious: spikes that align with reload events rather than traffic.
ML-based anomaly detection on memory and reload rate catches the change in growth slope (a team onboarding hundreds of new routes, a controller gone haywire) without you hand-tuning thresholds.
Goroutine and GC pause charts let you separate “table is too big” from “connections are leaking” in minutes, which determines whether the fix is config cleanup or timeout tuning.
Per-instance views make it straightforward to compare router counts, memory, and reload rates when you split into multiple Traefik deployments.