$ guides / traefik / traefik-gc-pause-latency ▌

Operations Guides

Traefik GC pauses: when Go garbage collection shows up in request latency

Your p99 latency graph has periodic spikes that do not line up with traffic, deployments, or backend slowness. The spikes hit every service at once, last tens to hundreds of milliseconds, and then everything goes back to normal. When a latency event affects all concurrent requests simultaneously and leaves backends untouched, the suspect list gets short, and Go garbage collection is near the top of it.

Traefik is a Go process. Every request it proxies runs on goroutines inside one shared heap, and when the runtime stops the world to collect garbage, every in-flight request waits. Most of the time these pauses are sub-millisecond and invisible. Under high heap size or a high allocation rate, they become long enough to show up in your request duration histograms.

Attribution first, tuning second. GC pauses are a diagnostic signal, not a primary alert. The goal is to confirm or rule out GC as the cause of a latency spike, find what is driving the allocations, and only then decide whether a runtime knob is the right fix.

What this means

Go’s garbage collector reclaims heap memory that is no longer referenced. Collection involves stop-the-world phases during which all goroutines are paused. For a reverse proxy, a pause is uniquely damaging because it is process-wide: a 50ms GC pause does not slow down one request, it adds 50ms to every request in flight at that moment.

Two conditions make pauses long enough to matter:

High heap size. The bigger the live heap, the more work each collection cycle does. Heuristic: GC pause p99 above 10ms means the collector is struggling with heap size and warrants investigation.
High allocation rate. Even with a modest heap, a workload that allocates aggressively forces frequent collections. In Traefik, the classic case is a configuration rebuild storm: every provider change rebuilds the routing table, allocating a fresh set of router, middleware, and service objects each time.

Go’s default GC target (GOGC=100) triggers a collection when the heap reaches roughly double the live heap from the previous cycle. Steady-state memory therefore sits around 2x live data, which is normal and not a leak. The failure shape you care about is different: pause durations climbing, collection frequency climbing, or both, correlated with latency spikes.

One more thing before tuning anything: Go memory pressure is graceful, then sudden. GC works harder and harder, pauses lengthen, and then the kernel OOM killer ends the process with no further warning. A rising GC pause trend on a growing heap is often the last observable signal before an OOM kill.

flowchart TD
    A[Latency spike across all services] --> B{Spike hits all concurrent requests?}
    B -- yes --> C{go_gc_duration_seconds p99 elevated?}
    B -- no --> D[Suspect single backend or middleware]
    C -- yes --> E{Heap growing or allocation rate high?}
    C -- no --> F[GC unlikely; check CPU throttle or access log blocking]
    E -- growing heap --> G[Hunt leak: goroutines, metric cardinality]
    E -- allocation storm --> H[Check config reload rate and churn]

Common causes

Cause	What it looks like	First thing to check
Config rebuild storm	`traefik_config_reloads_total` rate near or above 1/sec, CPU elevated, intermittent p99 spikes during mass deployments or pod churn	Reload rate vs. pause timestamps
Heap growth from routing table size	Memory climbs with router/service count, pauses lengthen gradually over weeks	`go_memstats_heap_inuse_bytes` trend vs. route count
Goroutine or connection leak	`go_goroutines` climbs monotonically without matching traffic; heap grows in lockstep; pauses lengthen before OOM	Goroutine count vs. request rate
Buffering middleware under load	Memory grows with request/response body sizes times concurrency; pauses correlate with traffic peaks	Which routers use buffering or compression middlewares
Prometheus metric cardinality growth	Heap grows as label cardinality grows; pauses worsen after enabling router-level labels or after 404 floods with unique paths	Heap growth after a metrics config change
Nothing wrong; GC is a red herring	Pauses under a few ms, spikes actually caused by access log buffer blocking or CPU throttling	Entrypoint vs. service latency split

Quick checks

All read-only. The metrics endpoint is typically on the dashboard entrypoint (port 8080 in many deployments).

# Pull GC pause quantiles
curl -s http://localhost:8080/metrics | grep go_gc_duration_seconds

# Heap in use and resident memory
curl -s http://localhost:8080/metrics | grep go_memstats_heap_inuse_bytes
grep VmRSS /proc/$(pgrep traefik)/status

# Goroutine count (leak check)
curl -s http://localhost:8080/metrics | grep go_goroutines

# Config reload rate (allocation storm check)
curl -s http://localhost:8080/metrics | grep traefik_config_reloads_total

# Service latency for correlation
curl -s http://localhost:8080/metrics | grep traefik_service_request_duration_seconds

Two samples of traefik_config_reloads_total 60 seconds apart give you reloads per second. Sustained rates near 1/sec are rebuild storm territory.

For per-cycle ground truth, running the process with GODEBUG=gctrace=1 logs every GC cycle to stderr. This is a stable Go runtime feature and works on any Go binary, but it requires an environment change and a process restart, so it is not a first move in a live incident.

How to diagnose it

Confirm the spike is process-wide. Compare traefik_entrypoint_request_duration_seconds against traefik_service_request_duration_seconds. GC pauses inflate both, for all services, at the same timestamps. A spike isolated to one service points at the backend instead.
Read the pause quantiles. go_gc_duration_seconds is a summary with quantiles from the standard Go collector. Sub-millisecond p99: GC is not your problem. p99 above roughly 10ms: investigate. p99 at 100ms or more sustained: ticket-worthy.
Line up the timestamps. Overlay pause duration with the latency spikes. GC attribution requires coincidence in time, not just both metrics being “high sometimes.”
Separate the two drivers. Is the heap large, or is the allocation rate high? A large, stable heap with lengthening pauses points to heap size (big routing table, accumulated metric series, a leak). A modest heap with frequent collections points to allocation rate (config churn).
Check the reload rate. If pause clusters line up with bursts in traefik_config_reloads_total, the routing table rebuild is the allocator. Correlate with whatever is churning: pod events, container start/stop, file provider writes.
Rule out a leak before tuning. If go_goroutines and heap climb monotonically over hours or days without matching traffic, you have a leak and no GC knob will save you. Capture a goroutine dump if the pprof endpoint is enabled: curl http://localhost:8080/debug/pprof/goroutine?debug=1. Identify what goroutines are waiting on; hung backend connections are the usual cause.
Rule out the impersonators. Two edge cases mimic GC: access log buffer blocking (entrypoint latency high, service latency normal, under extreme log volume) and CPU saturation from TLS handshakes or rebuilds. Check both before concluding GC.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`go_gc_duration_seconds`	Direct pause measurement; the attribution signal	p99 > 10ms warrants investigation; > 100ms sustained is ticket-worthy
`go_memstats_heap_inuse_bytes`	Live heap size; the input to pause duration	Growth without traffic growth; approaching container limit
`process_resident_memory_bytes`	What the OOM killer sees	Above ~60-80% of container limit sustained
`go_goroutines`	Leak detection; each goroutine pins heap	> 2-3x baseline without traffic increase
`traefik_config_reloads_total`	Allocation storm detector	Rate near or above 1/sec sustained
`traefik_service_request_duration_seconds`	Where the pause becomes user-visible	p99 spikes coinciding with pause spikes
`process_cpu_seconds_total`	GC competes with request serving for CPU	CPU elevated in step with reload rate or pause frequency

Fixes

Config rebuild storm

Increase providersThrottleDuration from its 2-second default to 5-10 seconds in busy environments so provider events batch into fewer rebuilds. Find the churn source: flapping health checks, aggressive autoscalers, or CI/CD deploying many services at once. Batch mass deployments where you can. This fixes the allocation rate, which is the actual problem; GC tuning would only move the pain around.

Heap too large

Reduce what the heap holds. Prune dead routers and services. If you enabled addRoutersLabels, weigh the cardinality cost against its value and consider dropping it. Reduce buffering middleware scope so fewer requests pin large body buffers. If the heap is legitimately large (tens of thousands of routes), consider splitting configuration across multiple Traefik instances.

Goroutine or connection leak

Set sane backend transport timeouts (dialTimeout, responseHeaderTimeout, idleConnTimeout in serversTransport) so requests to hung backends terminate instead of pinning goroutines forever. Identify and fix the misbehaving backend. Tuning GC against a leak only delays the OOM.

Runtime tuning, only after the above

If the heap is legitimately large, allocation rate is reasonable, and pauses are still hurting tail latency, two Go runtime knobs exist:

GOGC (default 100): lowering it collects more often against a smaller heap, trading CPU for shorter and more frequent cycles. Raising it does the opposite.
GOMEMLIMIT (Go 1.19+): a soft memory limit that makes the collector more aggressive as usage approaches it. In containers, set it to roughly 90% of the container memory limit, leaving room for non-heap memory.

Check whether your Traefik release sets either of these itself before overriding. Also rule out version-specific memory bugs before attributing growth to normal GC behavior; a confirmed leak in the compress middleware affected some Traefik v3 releases.

Do not restart Traefik as a “fix” for GC pauses. A restart resets the heap but changes nothing about what filled it.

Prevention

Alert on the trend, not the event. Track pause p99 and heap trend over days. GC pressure is one of the few OOM precursors you can actually watch.
Size memory with GC headroom. Set the container memory limit around 2x observed stable peak heap. Go’s collector needs that room.
Throttle providers by default in dynamic environments. Set providersThrottleDuration deliberately instead of inheriting the default during your next mass deployment.
Watch cardinality. Metric label growth is heap growth. Treat label-bearing config changes as capacity changes.
Keep the pause metric in your latency attribution workflow. It is low-severity as an alert and high-value as an explanation. When p99 spikes, pause quantiles should be one of the first three things you check, alongside the entrypoint/service latency split and the reload rate.

How Netdata helps

Per-second Go runtime metrics including GC pause duration, heap in use, and goroutines, collected alongside Traefik’s own request metrics, so pause events and latency spikes share a timeline.
Correlation in one view: overlay go_gc_duration_seconds with service request duration and config reload rate to confirm or eliminate GC attribution in seconds instead of scraping three endpoints by hand.
Leak detection: goroutine count and heap trends rendered over hours to days make monotonic, traffic-independent growth obvious long before the OOM kill.
Rebuild storm visibility: config reload rate graphed next to CPU and pause duration exposes the allocation-storm pattern that static dashboards miss.
Anomaly detection on latency percentiles that flags p99 spikes deviating from baseline even when they sit below your static SLO threshold.