process_resident_memory_bytes on your Traefik instance has been climbing for days. Request rates look normal, error rates are flat, and /ping returns 200. The question is whether the proxy legitimately needs more memory because its routing table grew, or whether it is leaking and will get OOM-killed at 3 a.m. with zero warning.

The failure mode is asymmetric. Go’s garbage collector absorbs growing allocation pressure gracefully, running more often and burning more CPU, until it cannot. Then the kernel OOM killer terminates the process instantly, dropping every connection in flight. There is no graceful degradation phase.

This guide separates the two cases: memory that tracks configuration size and traffic (expected), and memory that grows disconnected from both (a leak). The single most useful discriminator is the correlation between go_memstats_heap_inuse_bytes and go_goroutines.

What this means

Traefik’s memory footprint has three main drivers, each with a different trend signature:

  1. Routing table size. Every router, middleware, and service in the dynamic configuration consumes heap. Traefik v2 and later use significantly more memory per route than v1 did, due to the middleware chain architecture. Large routing tables (10,000+ routes) can consume hundreds of MB at rest. If you migrated from v2 to v3 and memory jumped, or if your route count is growing, this is likely the explanation. See the hub page’s failure pattern catalogue for where “routing table explosion” sits among Traefik’s failure archetypes.

  2. Connection and buffering state. One goroutine per active connection, plus middleware buffers. Middlewares that buffer request or response bodies (buffering, compression) consume memory proportional to body size times concurrent requests. Long-lived WebSocket and gRPC connections hold goroutines and buffers indefinitely.

  3. Metric cardinality. Prometheus metrics with per-router labels (addRoutersLabels: true) or high-cardinality header labels grow the in-memory metric store. Every unique label combination is a live series that is never freed.

A fourth category is genuine leaks: goroutines that never terminate (hung backend connections with no timeout), known middleware bugs, or plugin caches that never release. These show the same upward RSS trend but a different internal signature, which is what the diagnosis section isolates.

flowchart TD
    A[RSS trending up over days] --> B{heap_inuse growing too?}
    B -- No --> C[Go has not returned memory to OS.
Watch, but not a leak.]
    B -- Yes --> D{Goroutines growing with it?}
    D -- Yes --> E[Goroutine leak.
Capture pprof goroutine dump.]
    D -- No --> F{Route count or reload rate growing?}
    F -- Yes --> G[Legitimate routing-table growth.
Capacity-plan, consider providersThrottleDuration.]
    F -- No --> H{Metric cardinality or middleware change?}
    H -- Yes --> I[Cardinality growth or middleware leak.
Check labels, known issues.]
    H -- No --> J[Capture heap profile with pprof.
Find the allocation site.]

Common causes

CauseWhat it looks likeFirst thing to check
Routing table growthheap_inuse tracks route count; goroutines stable; memory steps up on config reloadsRouter/service count via /api/http/routers, config reload rate
v2 to v3 migrationStep change in baseline memory after upgrade, then stableRoute count unchanged but baseline higher; known v3 behavior
Goroutine leakheap_inuse and go_goroutines rising together, disconnected from trafficgo_goroutines trend vs request rate; pprof goroutine dump
Compress middleware bug (v3.0-v3.2)Rapid unbounded growth under load, sometimes tens of MB to GBTraefik version; whether compress middleware is in use
Metric cardinalitySlow steady growth; correlates with number of distinct label valuesaddRoutersLabels, headerLabels config; metric series count
Body buffering middlewareMemory tracks traffic peaks, spiky, recovers partiallyWhether buffering/compress middlewares are enabled
Regexp-heavy dynamic routesGrowth with route churn; regexp compiler allocations dominate heap profilepprof heap profile showing regexp compilation
Long-lived connectionsGoroutines and memory track WebSocket/gRPC connection counttraefik_open_connections trend vs request rate

Quick checks

All read-only. Run them against the metrics endpoint and the process.

# Live heap vs mapped heap: heap_inuse is the number that matters
curl -s http://localhost:8080/metrics | grep -E 'go_memstats_heap_(inuse|sys|alloc)_bytes'

# Goroutine count: the leak discriminator
curl -s http://localhost:8080/metrics | grep go_goroutines

# Process RSS from the kernel's point of view
grep VmRSS /proc/$(pgrep traefik)/status

# Config churn: is the routing table being rebuilt constantly?
curl -s http://localhost:8080/metrics | grep traefik_config_reload

# Connection load: are connections accumulating?
curl -s http://localhost:8080/metrics | grep traefik_open_connections

Two interpretive rules before you go further:

  • Use heap_inuse, not heap_sys. go_memstats_heap_sys_bytes includes memory the Go runtime has mapped from the OS but is not currently using. Go does not promptly return memory to the OS, so heap_sys and RSS can look alarming while live data is flat. go_memstats_heap_inuse_bytes is live allocations: if it is flat, you do not have a leak, you have a Go runtime holding onto freed pages.
  • Expect RSS to sit around 2x live data. With the default GOGC=100, GC triggers when the heap doubles. Steady-state RSS of roughly twice the live heap is normal Go behavior, not a leak.

How to diagnose it

  1. Establish the trend on heap_inuse, not RSS. Look at go_memstats_heap_inuse_bytes over 24-72 hours. Flat with a sawtooth (GC cycles) is healthy. A rising floor after each GC cycle is growth. A monotonic climb is a leak until proven otherwise.

  2. Correlate with goroutines. Plot go_goroutines over the same window. If goroutines and heap_inuse rise together while request rate and open connections are flat, you have a goroutine leak: each leaked goroutine pins its stack and every heap object reachable from its closure. This is the most common true leak pattern in a Go proxy.

  3. Correlate with routing table size. If goroutines are stable, check whether configuration is growing. Pull the current router and service inventory:

# How many routers and services are actually loaded?
curl -s http://localhost:8080/api/http/routers | jq 'length'
curl -s http://localhost:8080/api/http/services | jq 'length'

Compare with historical values if you have them. Also check the traefik_config_reloads_total rate: frequent rebuilds in a churning environment allocate a new router/middleware/service object graph each time, and memory steps up with each rebuild if the table is growing. Rebuild CPU cost scales with routers times middlewares times services.

  1. Check the version against known leaks. If you run Traefik v3.0 through v3.2 with the compress middleware enabled, you are exposed to a confirmed unbounded-growth bug in the compression algorithm priority, fixed in v3.3 (upstream issue #10859, fix PR #11641). Operators reported memory climbing from tens of MB to multiple GB under load with small route counts. If this matches your setup, upgrading or removing the compress middleware is the fix, not capacity planning. There are also upstream reports of regexp-heavy dynamic routing (issue #8044) and per-instance plugin caches (issue #11979) accumulating memory.

  2. Check metric cardinality. If addRoutersLabels: true is set, every router adds label combinations to every metric. If headerLabels maps a high-cardinality header (User-Agent, request IDs) into metric labels, series count grows unboundedly and each series is resident memory. Note that the Host header is promoted to Request.Host by Go and never appears in the header map, so headerLabels on Host silently does nothing; use X-Forwarded-Host if that is what you intended.

  3. Capture a heap profile. If none of the above explains it, get the allocation site directly. Enable api.debug: true in the static configuration, then:

# Capture a heap profile (requires api.debug: true; served on the API/dashboard entrypoint)
go tool pprof http://localhost:8080/debug/pprof/heap

Look at what dominates inuse_space. TLS connection buffers (crypto/tls internals) point at long-lived connections. Regexp compilation internals point at route churn with regexp rules. Middleware buffers point at buffering under traffic. Enabling the debug API exposes pprof endpoints; bind them to a private interface and do not leave them reachable from untrusted networks.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
go_memstats_heap_inuse_bytesLive heap allocations; the real leak indicatorRising floor after each GC cycle over days
process_resident_memory_bytesWhat the kernel and the OOM killer see> 80% of container limit, sustained
go_goroutinesDistinguishes goroutine leak from data growthRising with heap, disconnected from traffic
go_gc_duration_secondsGC struggling before the crashp99 pause time climbing steadily week over week
traefik_config_reloads_total rateConfig churn drives allocation and rebuild cost> 1/second sustained
Router/service count (via API)Direct measure of routing table sizeGrowth without a planned cause
traefik_open_connectionsConnection accumulation holds goroutines and buffersRising without corresponding request rate

The degradation curve: GC pause latency climbs first, then OOM arrives with no further warning. If heap_inuse exceeds roughly 80% of the container memory limit, runway is minutes to seconds depending on allocation rate. Size container memory limits at about 2x the observed stable peak heap so the GC has headroom and traffic spikes are absorbable.

Fixes

Legitimate routing-table growth

If the routing table is genuinely that big, the fix is capacity and architecture, not debugging. Raise the container memory limit to 2x peak heap. Reduce rebuild pressure with providersThrottleDuration (default 2s in v2; batching provider events reduces both rebuild CPU and allocation churn). If a single instance’s routing table is pushing hundreds of MB and rebuilds are visible in latency, consider sharding: split route ownership across multiple Traefik instances by namespace, domain, or entrypoint.

Goroutine leak

Identify what the leaked goroutines are blocked on from a goroutine profile (/debug/pprof/goroutine?debug=1). The usual root cause is a backend that accepts TCP connections but never responds, combined with missing or overly long transport timeouts. Set dialTimeout, responseHeaderTimeout, and idleConnTimeout under serversTransport.forwardingTimeouts so stuck requests terminate. Remove the misbehaving backend from rotation while you fix it. Restarting Traefik clears the accumulated goroutines but does not fix the cause; treat it as buying runway only.

Compress middleware bug

Upgrade to v3.3 or later, where the compression algorithm priority was reverted to the v2 behavior. If you cannot upgrade immediately, removing the compress middleware stops the growth at the cost of uncompressed responses.

Metric cardinality

Turn off addRoutersLabels unless you genuinely need per-router metrics. Audit headerLabels and drop any label sourced from a high-cardinality header. If Prometheus itself is also growing, the cardinality explosion is on both sides and scraping will get slow before Traefik runs out of memory.

Connection accumulation

If traefik_open_connections grows without request-rate growth, connections are not being closed. Check keep-alive alignment between Traefik and any upstream load balancer (the classic mismatch: the upstream LB’s idle timeout is longer than Traefik’s, so the LB reuses connections Traefik already closed). WebSocket-heavy workloads need a separate baseline; long-lived connections holding goroutines and buffers is expected behavior, and the fix is capacity, not timeouts you do not want.

Prevention

  • Alert on the pair, not the point. A static RSS threshold pages you too late or not at all. Alert on heap_inuse trend over 24h and on heap/goroutine divergence from traffic baseline. Anomaly detection on the ratio of memory growth to request-rate growth catches leaks weeks before the OOM.
  • Baseline after every upgrade. v2-to-v3 changed the per-route memory cost. Re-baseline heap, goroutines, and rebuild rate after any version upgrade before you declare it clean.
  • Cap config churn. Set providersThrottleDuration deliberately for your environment’s change rate, and investigate pathological provider churn (flapping health checks, autoscaler loops) rather than letting Traefik absorb it.
  • Set explicit transport timeouts. An unbounded responseHeaderTimeout is the single most common enabler of goroutine leaks. Every backend transport should have timeouts chosen deliberately.
  • Keep cardinality on a budget. Treat addRoutersLabels and headerLabels as spend decisions. If you enable per-router metrics, know your router count ceiling.
  • Watch GC pauses as the early warning. Rising go_gc_duration_seconds p99 means the heap is outgrowing the collector, and it appears before latency degrades and well before OOM.

How Netdata helps

  • Netdata charts go_memstats_heap_inuse_bytes alongside process_resident_memory_bytes per second, so you can see the post-GC floor rising (a leak) versus a flat floor under a high-water RSS (Go runtime behavior) without waiting for daily aggregates.
  • Overlaying go_goroutines on the heap trend makes the core discriminator in this article a visual check: the two lines moving together, disconnected from request rate, is the leak signature.
  • Config reload rate and open connections sit on the same dashboard, so the “growing routing table” versus “growing connection state” branches of the decision tree are one screen apart.
  • go_gc_duration_seconds percentiles surface the GC-struggle phase of the degradation curve, giving you a warning stage before the OOM cliff.
  • ML-based anomaly detection on the memory-to-traffic relationship flags slow, multi-day drift that static thresholds on absolute RSS routinely miss.