$ guides / traefik / traefik-goroutine-leak ▌

Operations Guides

Traefik goroutine leak: go_goroutines climbing toward OOM

Your Traefik dashboard shows go_goroutines climbing steadily over hours or days. Traffic is flat. Latency and error rates look normal. Yet the goroutine count keeps rising, RSS grows in lockstep, and you can extrapolate a straight line from today’s memory usage to the container limit and read off the day Traefik gets OOM-killed, dropping every in-flight connection with it.

In Traefik this almost always has the same shape: a backend accepts the TCP connection but never sends response headers, and the forwarding transport has no timeout (or a very long one) on that wait. Each stuck request leaves a goroutine blocked indefinitely. The goroutine count is the earliest visible signal, long before users feel anything.

This guide covers how to confirm the leak, identify which backend is causing it, and fix the timeout configuration that allowed it.

What this means

Traefik spawns goroutines per connection and per in-flight request (Go’s net/http model), plus a baseline set for provider watchers, health checkers, and internal loops. In steady state, the count is baseline goroutines (typically 50-200) plus active connections. After a traffic spike recedes, the count should fall back toward baseline within the keep-alive window. A count that climbs monotonically while request rate stays flat means goroutines are being created but never finishing.

The dominant cause: a backend completes the TCP handshake (so the dial succeeds and the connection looks healthy), then hangs before sending response headers. Traefik’s goroutine for that request blocks waiting for headers. If responseHeaderTimeout in the serversTransport is unset (it defaults to 0s, meaning unbounded, per the Traefik documentation) or set to minutes, the goroutine waits effectively forever. Repeat this for every request to the broken backend and goroutines accumulate.

Each leaked goroutine costs more than its ~8KB stack. It pins everything in its closure: request context, buffers, transport state. That is why process_resident_memory_bytes rises in lockstep with go_goroutines, and why the line does not bend until the process is killed.

flowchart TD
  A[Backend accepts TCP connection] --> B[Backend never sends response headers]
  B --> C[Request goroutine blocks with no or long responseHeaderTimeout]
  C --> D[go_goroutines climbs over hours to days]
  D --> E[Each goroutine pins stack and closure memory]
  E --> F[RSS grows in lockstep toward the container limit]
  F --> G[OOM kill: all connections dropped instantly]

The dangerous property of this failure mode is how healthy everything else looks. Latency and throughput stay normal until very late, because the leaked goroutines are parked, not consuming CPU. The only early signals are the goroutine count and memory trend themselves.

Common causes

Cause	What it looks like	First thing to check
Backend accepts TCP but never responds (deadlock, thread pool exhaustion) and `responseHeaderTimeout` is unset or very long	`go_goroutines` climbing, `traefik_open_connections` rising while request rate is flat	pprof goroutine dump: stacks blocked waiting for backend response
`responseHeaderTimeout` set but too long (e.g. 300s) under a sustained slow backend	Slower, sawtooth-ish growth; goroutines eventually expire but accumulate faster than they drain	Your `serversTransport` values vs. actual backend response times
WebSocket or long-lived connections without idle timeout	Connection count grows and never decays after traffic recedes	`traefik_open_connections` trend vs. request rate
Provider watcher or internal goroutine leak (rare, often version-specific)	Growth disconnected from any backend; stacks not in the forwarding path	pprof dump: what are the leaked goroutines actually doing
Plugin-related leak (middleware plugins interpreted at runtime)	Growth correlates with traffic through a specific plugin-enabled router	Disable the plugin on one router and compare growth rates

Quick checks

All read-only. Run them against the affected instance.

# Current goroutine count from the metrics endpoint
curl -s http://localhost:8080/metrics | grep go_goroutines

# Resident memory of the Traefik process
grep VmRSS /proc/$(pgrep traefik)/status

# Go heap in use (live allocations, not just RSS)
curl -s http://localhost:8080/metrics | grep go_memstats_heap_inuse_bytes

# Open connections per entrypoint (metric name varies by version and config:
# traefik_open_connections or traefik_entrypoint_open_connections)
curl -s http://localhost:8080/metrics | grep open_connections

# Entrypoint request rate for comparison (sample twice, 60s apart)
curl -s http://localhost:8080/metrics | grep traefik_entrypoint_requests_total

# Backend connection states held by the Traefik process
ss -tnp | grep traefik | awk '{print $1}' | sort | uniq -c

The decisive pattern: go_goroutines and RSS rising together while the traefik_entrypoint_requests_total rate is flat or falling. Rising goroutines plus rising request rate plus stable memory per goroutine is normal scaling, not a leak.

Two things that look like this but are not:

Startup spike. Traefik spawns provider watchers, health checkers, and internal loops simultaneously at boot; the count can jump to 200+ and settle. Growth that started at boot and then plateaued is normal.
Post-spike drain lag. After a real traffic spike, the count should return toward baseline within the keep-alive timeout. Only a count that never falls back is a leak.

How to diagnose it

Confirm the leak pattern. Plot or sample go_goroutines, process_resident_memory_bytes, and the entrypoint request rate over the same window. You want monotonic goroutine and RSS growth with flat traffic. If goroutines track traffic, stop here: you have a scaling question, not a leak.
Capture a goroutine profile. This requires the debug API (api.debug=true), which installs the pprof handlers:
```
# Dump all goroutine stacks in human-readable form
curl -s "http://localhost:8080/debug/pprof/goroutine?debug=1" > traefik-goroutines.txt
```
Note the security tradeoff: the debug and API endpoints expose internals, so enable them only on a loopback or restricted entrypoint, and disable them again when done.
Group the stacks. Most of the file will be the same stack repeated thousands of times. Count by signature:
```
# Rough histogram of what goroutines are waiting on
grep -A1 "^goroutine " traefik-goroutines.txt | grep -v "^--" | sort | uniq -c | sort -rn | head
```
A forwarding-side leak shows large counts of goroutines blocked in the net/http transport path, waiting on a response from a backend. If the dominant stacks are in a provider watcher, a middleware, or a plugin, that redirects the investigation.
Map the stuck stacks to a backend. From the stacks and from socket state (ss -tnp | grep traefik), identify which backend address the stuck connections point to. Cross-check with traefik_service_server_up: health checks may still pass for that backend if the health endpoint responds on a different code path than the hung application endpoints. This is the “healthy yet failing” case: server_up = 1, real requests hanging.
Check your timeout configuration. Look at the effective serversTransport for the affected service (via /api/rawdata or your dynamic config source). If responseHeaderTimeout is missing, it is unbounded. Also check dialTimeout and idleConnTimeout while you are there.
Decide severity. While goroutines are growing and RSS is well under the limit, this is a ticket: investigate during hours, fix the backend and the timeouts. When RSS crosses roughly 80% of the container limit, treat it as a page: the gap between “GC is struggling” and “OOM kill” is short, and the kill drops every connection at once.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`go_goroutines`	Earliest leak indicator; directly counts stuck work	Sustained >2-3x baseline with no matching traffic increase
`process_resident_memory_bytes`	Tracks what the OOM killer sees	Rising in lockstep with goroutines; >80% of container limit
`go_memstats_heap_inuse_bytes`	Live heap, confirms pinned memory rather than GC lag	Growing without corresponding traffic growth
`traefik_open_connections` (per entrypoint)	Distinguishes connection pile-up from pure goroutine growth	Rising while request rate is flat or falling
`traefik_entrypoint_requests_total` (rate)	The control variable: is traffic actually growing?	Flat or declining while goroutines rise confirms a leak
`traefik_service_server_up`	Locates the misbehaving backend, with the caveat that health checks can lie	Backend “up” while stuck connections pile up against it
`traefik_service_request_duration_seconds`	Stays normal until very late; a late-stage corroborating signal	p95 rising at the same time as goroutines means the leak is now affecting live traffic

Fixes

Remove the misbehaving backend from rotation

The backend that hangs after accepting connections is the root cause. Drain it, restart it, or pull it from the pool so new requests stop piling onto the leak. A restart of Traefik itself clears the accumulated goroutines and buys time, but it drops every in-flight connection instantly, and it does not fix anything: if the backend still hangs, the leak restarts from zero the moment traffic resumes. Fix the backend first; restart Traefik only when memory pressure forces it.

Set forwarding timeouts in serversTransport

Bound every stage of the backend interaction. In the dynamic configuration (v2/v3), define a serversTransport and attach it to the service:

# Dynamic configuration (file provider example)
http:
  serversTransports:
    guarded:
      forwardingTimeouts:
        dialTimeout: "30s"
        responseHeaderTimeout: "30s"
        idleConnTimeout: "90s"
  services:
    my-service:
      loadBalancer:
        serversTransport: guarded   # attach the transport to the service
        servers:
          - url: "http://backend:9000"

responseHeaderTimeout is the critical one for this failure mode. It caps how long Traefik waits for response headers after the request is fully written. With it set, a hung backend produces a 504 for that request and frees the goroutine instead of leaking it. Size it above your slowest legitimate response, not at zero and not at five minutes.
dialTimeout bounds connection establishment. It protects against backends that do not even complete the handshake.
idleConnTimeout reaps idle pooled connections so they cannot be silently killed by intermediaries and discovered dead later.

Tradeoffs: a responseHeaderTimeout that is too short will cut off legitimately slow endpoints (report generation, large exports) with 504s. Endpoints that legitimately stream or hold a response open need either a longer timeout on their specific serversTransport or a different handling strategy. Also remember the timeout chain: client, upstream load balancer, Traefik, backend. Each layer’s timeout should be longer than the one in front of it, or intermediaries will close connections that downstream components are still using, producing a different intermittent failure.

If the leak is not in the forwarding path

If pprof shows the leaked goroutines in a provider watcher, a middleware, or a plugin, timeouts will not help. Isolate by disabling the suspect component on one router or one instance and comparing growth rates. Check the Traefik issue tracker for your version before assuming your configuration is at fault.

Prevention

Never run a production service without responseHeaderTimeout. The default is unbounded, which converts any hung backend into a memory leak. Make a bounded serversTransport part of your service template.
Alert on the goroutine trend, not the absolute value. Baseline go_goroutines per instance and alert on sustained >2-3x baseline without a matching traffic increase. This catches the leak days before the OOM.
Alert on RSS versus the container limit. Goroutine alerts give you the trend; the memory alert gives you the deadline. RSS above 80% of the limit with a rising trend deserves immediate action.
Track open connections against request rate. A widening gap between traefik_open_connections and the request rate is an early connection-accumulation signal, upstream of the goroutine signal.
Rehearse the pprof capture. Enable the debug API on a restricted entrypoint before you need it, and confirm /debug/pprof/goroutine?debug=1 is reachable from your jump host. Mid-incident is the wrong time to learn the endpoint is not exposed.

How Netdata helps

Per-second go_goroutines and process RSS on the same timeline. The leak signature is two curves moving together while traffic is flat; seeing them overlaid at high resolution makes the divergence from traffic obvious in seconds.
Correlation with Traefik’s own metrics. Netdata’s Traefik collector pulls entrypoint request rates, response codes, and open connections alongside the Go runtime and process metrics, so the “goroutines up, traffic flat” comparison lives on one dashboard instead of two tools.
Container limit context. Netdata shows process memory against the cgroup limit, which turns “RSS is rising” into “RSS will hit the limit in roughly N hours at this rate.”
Trend alarms. Alerts on sustained goroutine growth and on memory-versus-limit ratios fire while the leak is still a ticket, not after the OOM kill makes it a page.