$ guides / traefik / traefik-high-latency ▌

Operations Guides

Traefik high request latency: isolating Traefik overhead from backend slowness

Latency dashboards are red, p95 is climbing, and the first question from the incident channel is “is it the proxy or the app?” With Traefik in the path, that question has a precise answer if you compare the right two metrics and read the JSON access-log fields most operators skip.

Traefik measures request duration at two points in its pipeline. traefik_entrypoint_request_duration_seconds measures the request at the edge, including TLS, middleware processing, and backend time. traefik_service_request_duration_seconds measures only the leg from Traefik to the backend and back. The difference between the two is Traefik’s own overhead. If entrypoint latency is high but service latency is normal, the time is being burned inside Traefik itself: TLS handshakes, middleware chain processing, gzip compression, response buffering, or an access-log write that is blocking request goroutines. If both are high, the backend is slow and Traefik is just the messenger.

This article walks through how to make that determination quickly, the gotchas that distort the histograms (streaming responses, retries, coarse default buckets), and how to fix each class of cause.

What this means

Every request through Traefik traverses: entrypoint (TLS termination, connection handling), router matching, middleware chain, service load balancer, backend. Only the first and last legs are instrumented with duration histograms. There are no per-middleware Prometheus metrics, so when overhead lives in the middleware chain you cannot see it directly. You infer it as a residual.

flowchart LR
    C[Client] -->|entrypoint timer starts| E[Entrypoint + TLS]
    E --> M[Middleware chain]
    M -->|service timer starts| S[Service LB]
    S --> B[Backend]
    B -->|service timer stops| S
    S --> M2[Response middlewares: gzip, buffering]
    M2 -->|entrypoint timer stops| C
    M2 -.->|access log write, can block| L[Access log]

The residual logic: entrypoint duration - service duration = Traefik overhead (TLS, router matching, middleware, response processing, logging). This comparison is the whole diagnostic. Everything else in this article is about making that comparison trustworthy.

Common causes

Cause	What it looks like	First thing to check
Backend slowness	Both entrypoint and service latency elevated together, per service	Backend CPU, DB query times, downstream dependencies for the affected service
TLS handshake CPU pressure	Entrypoint latency high, service latency normal, CPU high, spike in new TLS sessions	`process_cpu_seconds_total` rate vs `traefik_entrypoint_requests_tls_total` rate
Buffering or gzip middleware	Huge `Overhead` in JSON access logs relative to `OriginDuration`	Access log: compare `Overhead` vs `OriginDuration` per request
Access log buffer block	Entrypoint latency high, service latency normal, no obvious middleware cause	Access log destination health (disk full, blocked pipe); `bufferingSize` setting
Retry amplification	Latency rising together with `traefik_service_retries_total`	Retry rate vs request rate for the affected service
Streaming responses misread as overhead	Entrypoint latency far above service latency on SSE/download endpoints	Identify which routes are streaming; exclude them from the comparison
Connection pool churn to backends	Latency spikes on first requests, high TIME_WAIT to backends	`ss -tn state time-wait` counts, `MaxIdleConnsPerHost` config
GC pauses	Periodic latency spikes across all services at once	`go_gc_duration_seconds` p99, `process_resident_memory_bytes` trend

Quick checks

All read-only. Run from a host that can reach Traefik’s metrics endpoint (typically :8080/metrics) and its access log.

# 1. Compare entrypoint vs service duration histograms
curl -s http://localhost:8080/metrics | grep traefik_entrypoint_request_duration_seconds
curl -s http://localhost:8080/metrics | grep traefik_service_request_duration_seconds

# 2. Retry rate (amplification check)
curl -s http://localhost:8080/metrics | grep traefik_service_retries_total

# 3. Backend health (is latency from a shrinking pool?)
curl -s http://localhost:8080/metrics | grep traefik_service_server_up

# 4. CPU of the Traefik process (TLS/middleware pressure)
curl -s http://localhost:8080/metrics | grep process_cpu_seconds_total

# 5. TLS session rate (handshake storm?)
curl -s http://localhost:8080/metrics | grep traefik_entrypoint_requests_tls_total

# 6. Measure TLS handshake time from a client
curl -w "tls_handshake: %{time_appconnect} - tcp_connect: %{time_connect}\n" \
  -o /dev/null -s https://your-traefik-host/

# 7. Goroutine and GC health
curl -s http://localhost:8080/metrics | grep -E 'go_goroutines|go_gc_duration_seconds'

# 8. Backend connection states (pool churn, CLOSE_WAIT leak)
ss -tnp | grep traefik | awk '{print $1}' | sort | uniq -c

If access logs are in JSON format, the single most diagnostic query compares OriginDuration (time the backend took, in nanoseconds) against Overhead (time Traefik spent, in nanoseconds) per request:

# Find requests where Traefik overhead dwarfs backend time
jq 'select(.Overhead > .OriginDuration) | {path: .RequestPath, origin: .OriginDuration, overhead: .Overhead}' \
  /var/log/traefik/access.log | head -40

A request with OriginDuration of 22ms and Overhead of 2214ms is a smoking gun: the backend is fine, and over two seconds is being consumed inside Traefik’s own processing.

How to diagnose it

Scope the symptom to a service. Pull traefik_service_request_duration_seconds and find which service labels have shifted buckets. If all services degraded simultaneously, suspect a shared Traefik-level cause (CPU, GC, access log) rather than N independent backends failing at once.
Compare the two histograms for the same traffic. For the affected service, check whether traefik_entrypoint_request_duration_seconds shows the same shift as the service histogram. Same shift: backend problem. Entrypoint shifted but service flat: overhead is inside Traefik.
Confirm with access logs. For a sample of slow requests, compare Overhead to OriginDuration. This is the per-request version of step 2 and removes histogram-bucket ambiguity. High Overhead with low OriginDuration confirms Traefik-side overhead; the reverse confirms backend slowness.
If the overhead is inside Traefik, bisect the middleware chain. There are no per-middleware metrics, so work from the router’s middleware list. The usual suspects, in rough order of cost: response buffering middleware, gzip compression, regex-heavy routing or header manipulation, and ForwardAuth (which adds a synchronous network call per request; check the auth service’s own latency). Check the access log GzipRatio field if compression is in the chain.
Check the access log itself. If log volume is extreme (thousands of requests per second with verbose fields), the internal log buffer can fill and block request goroutines. The signature is entrypoint latency high, service latency normal, and no middleware explanation. Verify the log destination is healthy (disk not full, pipe not blocked) and whether bufferingSize is configured. A bufferingSize of 0 means synchronous writes on the request path.
Check for retry amplification. A rising traefik_service_retries_total concurrent with rising latency means Traefik is re-sending failed requests, multiplying backend load. Retries are measured individually in the duration histogram, so one logical client request appears as multiple samples, which also distorts your percentiles upward.
Rule out measurement artifacts before concluding anything. Streaming responses and coarse buckets regularly produce false “Traefik overhead” readings; see the next section.

Three ways the histograms lie to you

Streaming responses. For streaming endpoints (SSE, large file downloads, long-lived gRPC streams), the service-level histogram effectively records time-to-first-byte behavior while the entrypoint-level histogram reflects the full response lifetime. Entrypoint latency will look enormously higher than service latency, and the gap is not middleware overhead. It is the stream’s duration. Identify streaming routes and exclude them from the comparison or baseline them separately.

Retries. Each retry attempt is recorded as a separate duration sample. A request that failed once and succeeded on retry contributes two samples to the service histogram, one of them a failure-path duration. Under retry storms, histograms mix fast successes and slow failed attempts.

Coarse default buckets. The default histogram buckets are [0.1, 0.3, 1.2, 5.0] seconds. A service that degrades from 40ms to 90ms has doubled its latency, and both states land in the same le="0.1" bucket. The histogram cannot see the change. For any service with an SLA under ~100ms, configure finer buckets in the metrics configuration or rely on access-log fields for percentiles. Do not trust a p95 computed from four buckets that stop at 100ms granularity.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`traefik_service_request_duration_seconds`	Backend-side latency per service	Bucket distribution shifting upward for one service
`traefik_entrypoint_request_duration_seconds`	Total latency including Traefik processing	Elevated while service histogram is flat (overhead inside Traefik)
Access log `Overhead` vs `OriginDuration`	Per-request ground truth for the split	`Overhead` » `OriginDuration`
`traefik_service_retries_total`	Retry amplification inflating latency and load	Retry rate > 5% of request rate
`traefik_service_server_up`	Shrinking backend pool concentrates load	Any backend at 0; all at 0 means 503s are next
`process_cpu_seconds_total` (rate)	TLS handshakes and middleware are CPU-bound	Sustained high CPU correlated with latency
`traefik_entrypoint_requests_tls_total` (rate)	New TLS session rate; handshakes dominate CPU	Spike coinciding with entrypoint latency rise
`go_gc_duration_seconds`	GC pauses stall all in-flight requests	p99 pause > 100ms
`go_goroutines`	Hung backends hold goroutines and inflate latency later	Monotonic growth without traffic growth

Fixes

Backend is slow (both histograms high)

Traefik is not your problem. Check the backend’s resource utilization, database query times, connection pools, and downstream dependencies. Cross-reference traefik_service_server_up: if part of the pool is down, the survivors may simply be overloaded, and the fix is restoring capacity, not tuning Traefik. If the backend is alive but too slow, the failure mode and timeout tuning are covered in Traefik 504 Gateway Timeout: the backend is alive but too slow.

TLS handshake pressure

Switching certificate key types from RSA to ECDSA reduces handshake CPU cost substantially. Verify TLS session resumption is working so returning clients skip full handshakes. If CPU is saturated by handshakes at your traffic level, scale Traefik horizontally; handshake cost does not yield to configuration tuning.

Buffering and gzip middleware

Buffering middleware with large memory limits can add seconds of overhead per request on large bodies. If you need buffering, constrain memRequestBodyBytes / memResponseBodyBytes and set hard maxRequestBodyBytes limits rather than buffering unbounded bodies. For gzip, exclude already-compressed content types (images, video, archives) and very small responses from compression; the CPU cost is real and the ratio is poor. Use the access-log GzipRatio field to find routes where compression buys nothing.

Access log blocking

Set bufferingSize (for example 100) so log lines are buffered and written asynchronously instead of synchronously on the request path. Reduce volume with filters.statusCodes (log only errors) or filters.minDuration (log only slow requests) if you can tolerate the loss. Verify the destination disk or pipe is not full or blocked. Even with these mitigations, access logging has a real throughput cost at high request rates; measure before and after.

Retry amplification

Reduce retry attempts or remove the retry middleware from the affected router while the backend recovers. Retries on a degrading backend convert a partial failure into a total one; see Traefik cascading backend failure: how a partial outage becomes a total one for the full pattern. Longer term, scope retries to idempotent methods and alert on the retry-to-request ratio.

Fix the measurement

Configure histogram buckets appropriate to each service’s actual latency range instead of the defaults, and build per-service baselines. A single latency threshold across heterogeneous backends (compute API, static files, WebSocket, streaming) masks exactly the regressions you care about. A 200ms p95 is excellent for one service and a page-worthy regression for another.

Prevention

Per-service baselines and per-service alerts. Alert on deviation from each service’s own baseline, not a universal threshold. This is the single highest-leverage change for latency monitoring.
Buckets that match the SLA. If your p99 target is 150ms, buckets starting at 100ms are useless. Set explicit bucket lists in the metrics configuration at the granularity you need.
Keep Overhead and OriginDuration in the access log. JSON access logs with duration fields are the only per-request source of truth for the Traefik/backend split. There is no dedicated Prometheus metric for Traefik-induced latency.
Trend the retry ratio. traefik_service_retries_total rate divided by request rate should be a standing dashboard panel. Latency incidents preceded by rising retries are the amplification pattern, and catching it early is the difference between a slow service and a down one.
Watch config reload churn. If latency spikes correlate with traefik_config_reloads_total increments, routing-table rebuilds are competing with request serving. Increase providersThrottleDuration in busy dynamic environments.
Load-test with access logging on. Access log cost is invisible until production volume. Benchmark your log configuration at realistic request rates, including the failure case of a slow or blocked destination.

How Netdata helps

Netdata charts traefik_service_request_duration_seconds and traefik_entrypoint_request_duration_seconds side by side per service and entrypoint, so the overhead residual is visible without writing PromQL during an incident.
Retry rate (traefik_service_retries_total) plotted against request rate and latency makes the amplification pattern obvious as three lines converging.
traefik_service_server_up per backend URL shows whether latency is coming from a shrinking pool rather than a uniformly slow application.
Go runtime signals (go_goroutines, go_gc_duration_seconds, process_resident_memory_bytes) are collected alongside Traefik metrics, so GC pauses and goroutine growth can be correlated with latency spikes on one dashboard.
Process CPU and TLS session rate correlation helps distinguish handshake-bound overhead from middleware-bound overhead.
Per-service anomaly detection on latency surfaces deviations from each service’s own baseline, which is the alerting model this symptom actually needs.