$ guides / traefik / traefik-out-of-memory-oom ▌

Operations Guides

Traefik out of memory: OOM kills and the crash loop that follows

Traefik was proxying traffic normally. Then the process vanished. In Kubernetes you see OOMKilled in the pod’s last terminated state and a restart count that keeps climbing. On bare metal or Docker you see exit code 137 and a container that came back up, ran for a while, and died again.

An OOM kill has no graceful degradation phase. Go’s garbage collector absorbs growing memory pressure by running more often, right up until the moment it cannot. Then the kernel OOM killer terminates the process instantly. Every in-flight connection drops at once: client connections, backend connections, WebSockets, gRPC streams. From the outside it looks like a total, simultaneous outage of everything behind the proxy.

The crash loop that follows makes diagnosis harder, not easier. Each restart clears the heap, so by the time you look at the process it is healthy and climbing toward the same wall. The evidence you need is in the growth curve before the kill, not in the process after it.

What this means

The kernel (or the container runtime enforcing the cgroup memory limit) killed the Traefik process because it ran out of memory. Two things follow:

The limit that matters is the cgroup limit, not host RAM. A Traefik container with a 1 GiB limit on a 64 GiB host dies at 1 GiB. Comparing process_resident_memory_bytes against total system memory tells you everything is fine right up to the kill.
RSS is not live data. With the default GOGC=100, the Go runtime triggers GC when the heap doubles, so steady-state RSS is roughly 2x the live heap. A process whose live data is 400 MiB will sit around 800 MiB of RSS. This is normal, and it is why headroom calculations go wrong when teams size the container limit to observed RSS without understanding the multiplier.

flowchart TD
  A[Memory driver: leak, buffering, routing table, log buffer] --> B[RSS climbs toward cgroup limit]
  B --> C{GC keeping up?}
  C -->|yes| D[Longer GC pauses, latency jitter]
  D --> B
  C -->|no| E[Kernel OOM kill: instant, all connections dropped]
  E --> F[Restart: heap cleared, process healthy]
  F --> G[Traffic returns, memory climbs again]
  G --> B

Each trip around that loop drops all connections and produces a burst of client-visible errors. The loop period is your diagnostic window: headroom divided by growth rate tells you roughly how long each cycle takes.

Common causes

Cause	What it looks like	First thing to check
Goroutine leak (hung backend connections)	`go_goroutines` climbs monotonically, disconnected from traffic; RSS climbs in lockstep	Goroutine profile from pprof; look for goroutines blocked on backend reads
Request/response body-buffering middleware	Memory tracks concurrent request count and payload size; spikes under large uploads/downloads	Which routers have buffering or compress middlewares attached
Access log buffer growth	Entrypoint latency rises while service latency stays normal; memory grows when log output is slow or blocked	Whether the log destination (disk, pipe) is keeping up
Routing table size	High baseline memory at rest that scales with router/service count; grows after onboarding waves	Router and service counts in the provider; memory per config reload
Config-object accumulation across reloads	Stepwise memory increases correlated with `traefik_config_reloads_total` increments	Overlay reload count onto the RSS curve
Metric cardinality (per-router labels)	Memory growth after enabling `addRoutersLabels`; series count explodes	Cardinality of the metrics endpoint

Two version-specific notes. Traefik v3.0 through v3.2 had a memory leak in the compress middleware (the brotli encoder consumed memory disproportionately under load); this was fixed in v3.3 by reprioritizing compression algorithms. Operators on affected v3 versions reported memory returning to normal after removing the compress middleware. Separately, one operator reported that the file provider with watch: true correlated with their leak and that watch: false resolved it; this was never confirmed as a core bug.

Quick checks

All read-only. Run these against the host and the metrics endpoint (adjust the port and pid for your deployment).

# 1. Confirm the kill actually was OOM (host kernel log)
dmesg -T | grep -i -E 'oom|killed process' | tail -20

# 2. In Kubernetes: confirm OOMKilled and see the restart count
kubectl get pod -n <ns> <traefik-pod> -o jsonpath='{.status.containerStatuses[*].lastState.terminated}{"\n"}'
kubectl get pod -n <ns> <traefik-pod> -o jsonpath='{.status.containerStatuses[*].restartCount}{"\n"}'

# 3. In Docker: check the last exit state (137 = 128 + SIGKILL)
docker inspect <container> --format '{{.State.ExitCode}} {{.State.OOMKilled}} {{.State.Restarting}}'

# 4. Current RSS of the live process
grep VmRSS /proc/$(pgrep -x traefik)/status

# 5. The limit that actually applies (cgroup, not host RAM)
cat /sys/fs/cgroup/memory.max 2>/dev/null || cat /sys/fs/cgroup/memory/memory.limit_in_bytes

# 6. Goroutine count right now
curl -s http://localhost:8080/metrics | grep '^go_goroutines'

# 7. Heap in use (live data) vs RSS
curl -s http://localhost:8080/metrics | grep -E '^go_memstats_heap_inuse_bytes|^process_resident_memory_bytes'

# 8. GC pause behavior
curl -s http://localhost:8080/metrics | grep '^go_gc_duration_seconds'

# 9. Config reload rate (is memory growth tracking reloads?)
curl -s http://localhost:8080/metrics | grep '^traefik_config_reloads_total'

Note on the cgroup path: v2 uses memory.max, v1 uses memory/memory.limit_in_bytes. A value of max or a very large number means no limit is set, which shifts the question to host-level memory pressure.

How to diagnose it

The goal is to separate three cases: a leak (unbounded growth, needs a code or config fix), normal scaling (growth proportional to load, needs more headroom), and GC arithmetic (the process is fine but the limit is too tight for Go’s 2x RSS behavior).

Confirm the OOM and find the limit. Steps 1-3 above confirm the kill; step 5 gives you the number that matters. Everything else is measured against that number.
Reconstruct the growth curve. You need the RSS trend before the kill, which means a metrics system with history, not the live process. Plot process_resident_memory_bytes over the hours before the last few restarts. Three shapes:
- Monotonic climb over hours/days, disconnected from traffic: leak. Continue to step 3.
- Climb that tracks request rate and connection count, flattening off-peak: normal scaling or buffering proportional to load. Jump to step 5.
- Sawtooth that resets at each restart and climbs again at the same rate: either, but the restart period gives you the growth rate: (limit minus baseline) divided by cycle time.
Correlate with goroutines. Overlay go_goroutines on the same window. If goroutines and RSS climb together while request rate is flat, you have the classic goroutine leak: something (usually a backend that accepts connections and never responds, with no effective timeout) is pinning goroutines, and each leaked goroutine holds its closure’s memory. This is the most common Traefik OOM mechanism.
Capture a goroutine profile before the next kill. With the debug API enabled, pull the profile while memory is high:
```
# Requires the debug/pprof endpoint to be enabled (--api.debug=true)
curl -s 'http://localhost:8080/debug/pprof/goroutine?debug=1' > goroutines.txt
```
Group the stacks. Thousands of goroutines parked in the same backend-read or transport stack point at the offending backend and at missing timeouts (dialTimeout, responseHeaderTimeout, idleConnTimeout in serversTransport). Do not enable the debug API on a publicly reachable entrypoint; it exposes profiling data about your infrastructure.
Check the middleware chain for buffering. Buffering and compression middlewares hold memory proportional to request/response body size times concurrency. If growth tracks large request or response bodies (check traefik_service_requests_bytes_total and traefik_service_responses_bytes_total alongside RSS), audit which routers carry buffering or compress middlewares, and on Traefik v3.0-3.2 treat the compress middleware as a prime suspect.
Check reload correlation and routing-table size. Overlay traefik_config_reloads_total increments onto RSS. Stepwise increases after reloads in a high-churn environment suggest config-object accumulation. A high memory floor at rest that scales with router count is routing-table cost; very large routing tables (thousands of routers) can consume hundreds of MB before a single request arrives.
Rule out log-buffer growth. If entrypoint latency rose while service latency stayed flat in the same window, suspect the access log writer blocking on a slow or full destination; blocked log goroutines accumulate memory the same way hung backend goroutines do. Check the filesystem the access log writes to.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`process_resident_memory_bytes` vs cgroup limit	The actual OOM distance	Sustained above ~80% of the limit, or rising trend
`go_memstats_heap_inuse_bytes`	Live heap; separates real growth from GC headroom	Growing without corresponding traffic growth
`go_goroutines`	Earliest leak indicator; each goroutine pins memory	Sustained growth without matching connection/request growth
`go_gc_duration_seconds`	GC struggling is the last warning before the wall	p99 pauses climbing as RSS approaches the limit
`traefik_config_reloads_total`	Links memory steps to config churn	Reload rate high and RSS stepping up with it
`process_start_time_seconds` / restart count	Detects the crash loop itself	Multiple restarts within 30 minutes
`traefik_open_connections`	Connections rising without request-rate growth feeds both FD and memory pressure	Steady climb with flat request rate

Alert on the ratio of RSS to the container limit, not the absolute value. Memory above 80% of the limit is a ticket; the reason it cannot be a page from metrics alone is that the monitoring system often does not know the cgroup limit. Fix that gap explicitly: record the limit (or read it from the cgroup) and alert on the ratio, because by the time the OOM kill fires you have already lost every connection.

Fixes

Goroutine leak from hung backends

Set explicit transport timeouts in serversTransport so no request can pin a goroutine indefinitely: dialTimeout, responseHeaderTimeout, and idleConnTimeout. The specific failure this addresses is a backend that accepts the TCP connection but never responds; without responseHeaderTimeout, Traefik waits forever and each such request is a permanent goroutine. Identify the offending backend from the goroutine profile and remove it from rotation while you fix it. Tradeoff: timeouts that are too aggressive produce 504s for legitimately slow backends, so size them from observed p99 service latency, not guesses. See Traefik 504 Gateway Timeout.

Buffering and compression middleware

Remove buffering middlewares from routers that do not need them, and avoid buffering in front of large-payload routes (file upload/download, media). On Traefik v3.0-3.2, upgrade to v3.3 or later for the compress middleware fix; if you cannot upgrade immediately, removing the compress middleware is the reported workaround. Tradeoff: dropping compression increases egress bandwidth and response times for compressible content; dropping buffering removes its protection for slow backends.

Access log pressure

Point access logs at stdout or an async destination, reduce verbosity (drop fields you do not use), and make sure the destination filesystem has space and write throughput. Tradeoff: less log detail for forensics; mitigate by keeping full detail on error responses only if your log format supports it.

Routing table and config churn

Consolidate routers where rules can be merged, and split very large configurations across multiple Traefik instances by responsibility. In high-churn environments, raise providersThrottleDuration so bursts of provider events batch into fewer rebuilds; this reduces both rebuild CPU and the allocation churn that feeds GC pressure. Tradeoff: longer throttle means slower convergence when you legitimately deploy.

Headroom and runtime limits

Size the container memory limit at roughly 2x the observed stable peak heap, per the GC arithmetic above: live data plus one doubling. On Go 1.19 and later, set GOMEMLIMIT below the cgroup limit so the runtime GCs harder as it approaches the wall instead of coasting into it; this converts some OOM kills into elevated GC CPU, which is survivable and visible. Tradeoff: aggressive GC costs CPU and adds latency jitter, which beats an instant kill but is not free.

Prevention

Alert on the ratio, not the absolute. RSS against the cgroup limit, with a ticket at ~80% sustained. A dead proxy is a page; a climbing ratio is the ticket that prevents it.
Trend goroutines against traffic. go_goroutines growing faster than connections or requests is the cheapest early-warning signal you have. Baseline it once.
Set transport timeouts before you need them. Every Traefik in production should have explicit dialTimeout, responseHeaderTimeout, and idleConnTimeout. Their absence is what turns one wedged backend into a leaked-goroutine farm.
Restart-count alerting. Multiple restarts in 30 minutes is the crash-loop signature. Catch the second kill, not the tenth.
Audit middleware chains after changes. Buffering and compression middlewares are memory multipliers; review which routers carry them during change review.
Load-test at production concurrency before raising traffic. Buffering and goroutine counts scale with concurrency, not request rate, so memory behavior at 10x your test concurrency is not linear.

How Netdata helps

Per-second RSS and Go runtime metrics on the Traefik process show the exact growth curve before each kill, which is the evidence the post-restart process no longer has.
Goroutine count alongside connection and request rates makes the leak-versus-scaling distinction visible on one screen instead of three terminals.
GC pause duration correlated with request latency shows when GC pressure is degrading traffic before the process dies, giving you an earlier tripwire than the OOM itself.
Container memory usage versus the cgroup limit is collected together, so the ratio that matters is computed from the right denominator rather than host RAM.
Restart events overlaid on memory and traffic turn the crash loop into a readable cycle: climb, kill, restart, climb, with the growth rate measurable from the chart.