Traefik entrypoint vs service latency: measuring middleware overhead

A client reports that requests through Traefik are slow. You check traefik_service_request_duration_seconds and the backends look fine: p95 is 80ms, well inside the SLO. Yet clients measure 400ms end to end. The missing time is being spent inside Traefik itself, and the service-level metric cannot see it.

Traefik exposes request duration at two points in its pipeline: at the entrypoint, where the request arrives from the client, and at the service, where the request is handed to a backend. The difference between the two is Traefik’s own overhead: TLS termination, router matching, the middleware chain, compression, and request/response buffering. Because Traefik ships no per-middleware Prometheus metrics, this subtraction is the only metrics-based way to localise middleware cost without capturing Go profiles or traces.

This article covers what each metric actually measures, how to compute the gap in PromQL, what a large gap means, and the pitfalls that make the numbers lie.

What the two metrics measure

traefik_entrypoint_request_duration_seconds is a histogram with labels code, method, protocol, and entrypoint. It measures the full round trip as Traefik sees it: from receiving the request on the listener to sending the response back to the client. That includes TLS handshake effects on the connection, router evaluation, every middleware in the chain, and the backend call itself.

traefik_service_request_duration_seconds is a histogram with labels code, method, protocol, and service. It measures only the time spent communicating with the backend: network transit to the upstream, backend processing, and response transfer back into Traefik. It does not include client-side TLS or middleware processing that happens before the request reaches the service.

Both default to buckets of [0.1, 0.3, 1.2, 5.0] seconds, which matters later when you try to read percentiles off them.

flowchart LR
  C[Client] --> E[Entrypoint: TLS termination]
  E --> R[Router matching]
  R --> M[Middleware chain: auth, ratelimit, gzip, buffering]
  M --> S[Service: load balancer]
  S --> B[Backend]
  B --> S
  S --> M
  M --> E
  E --> C
  E -. "entrypoint duration: full round trip" .-> E
  S -. "service duration: backend call only" .-> S

The gap between the two measurements is everything above the service line: TLS, routing, middleware, and Traefik-internal blocking.

The subtraction technique

Compute both percentiles over the same window and subtract:

# p95 at the entrypoint (client-perceived, Traefik side)
histogram_quantile(0.95,
  sum(rate(traefik_entrypoint_request_duration_seconds_bucket[5m])) by (le))

# p95 at the service (backend side)
histogram_quantile(0.95,
  sum(rate(traefik_service_request_duration_seconds_bucket[5m])) by (le))

# Traefik's own overhead, per request
histogram_quantile(0.95,
  sum(rate(traefik_entrypoint_request_duration_seconds_bucket[5m])) by (le))
  -
histogram_quantile(0.95,
  sum(rate(traefik_service_request_duration_seconds_bucket[5m])) by (le))

Graph both lines and the derived gap rather than computing it once. The shape of the gap over time is what diagnoses the problem:

Gap large and growing while service latency is flat. The bottleneck is Traefik’s own stack: TLS handshakes, an expensive middleware (gzip on large responses, regex-heavy rules, ForwardAuth calls), or internal blocking such as a full access log buffer.
Gap small and both latencies high together. The backend is slow. Traefik is just the messenger. Investigate the upstream, not the proxy.
Gap large only on first requests after idle periods. TLS handshake cost for new connections. Check your new-connection rate and certificate type, not the middleware chain.

Subtracting two histogram_quantile results is an approximation. You are comparing percentile estimates derived from coarse buckets, not per-request timings. Treat the gap as a localisation tool (“which side of the line is the time going?”), not as an exact number of milliseconds attributable to a specific middleware.

What inflates the gap

Cause	What it looks like	First thing to check
TLS handshake pressure	Gap widens with new-connection rate; CPU climbs	`traefik_entrypoint_requests_tls_total` rate vs `process_cpu_seconds_total` rate
Compression middleware	Gap grows with response size; CPU high at moderate RPS	Which routers have gzip; response bytes via `traefik_service_responses_bytes_total`
Regex-heavy routing or middleware rules	CPU high even at low request rate; gap flat but large	Rule complexity in dynamic config; CPU profile if needed
Buffering middleware	Gap scales with payload size; memory climbs	`process_resident_memory_bytes` trend alongside payload sizes
Access log buffer blocking	Gap spikes with no upstream latency increase	Log volume, output destination health, disk saturation on the log path
ForwardAuth or external middleware	Gap tracks the auth service’s latency, not yours	Latency of the external auth endpoint

The access log case deserves emphasis because it is counterintuitive. When access log volume is extremely high and the output destination slows down, Traefik’s internal log buffer fills and request-handling goroutines block waiting to write. The symptom is exactly a large entrypoint-minus-service gap with completely normal backend latency. The fix is operational: switch to stdout/async logging, reduce verbosity, or fix the blocked destination. Do not go hunting through middleware config first.

Isolating further

The subtraction tells you the time is inside Traefik. It does not tell you which middleware. Your options, in order of increasing effort:

Bisect by router. Different routers carry different middleware chains. If addRoutersLabels is enabled, compare entrypoint-level latency across entrypoints and correlate with which routers serve the slow traffic. On Traefik v3, traefik_router_request_duration_seconds exists with router and service labels; Traefik documents it as a request-processing histogram on a router, not a clean split of routing from middleware overhead, so confirm attribution with detailed tracing or a config bisect.
Bisect by config. Remove middlewares from a router one at a time in a staging or canary route and watch the gap close. Crude but decisive, and safe if you use a dedicated test router rather than editing production chains.
Distributed tracing. Traefik v3.5 added a traceVerbosity option on entrypoints and routers. The default minimal produces one server span and one client span per request. Setting traceVerbosity: detailed emits per-middleware spans, which gives you the per-middleware timing that Prometheus metrics never will. This is the only way to attribute time to a specific middleware without a config bisect.
Go pprof. A CPU profile (/debug/pprof/profile, requires the debug API or pprof endpoint to be enabled) shows where CPU time goes: crypto/rsa for handshake-bound, flate/gzip for compression-bound, regexp for rule-bound. This confirms a hypothesis but does not give per-request timing.

Pitfalls that make the numbers lie

Coarse buckets. Default buckets top out at 5.0s and have nothing between 0.1 and 0.3s. If your services answer in 50ms, p95 estimates from these buckets are rough at best, and the gap between two rough estimates is rougher. Configure buckets that match your latency profile and your forwarding timeouts before drawing conclusions.
The buffering middleware code="0" bug. In early v2 releases, service-level metrics could record code="0" instead of the real status when the buffering middleware was in the chain, while entrypoint metrics recorded the correct code. This was fixed in Traefik v2.9.4 by PR #9440. If you run an affected old release and filter or group the subtraction by code, your two sides may not line up.
Retries inflate service-side measurements. Each retry attempt is measured individually, so a retried request appears as multiple service durations against one entrypoint duration. Check traefik_service_retries_total before trusting a small gap on a flaky service.
Streaming and long-lived responses. Server-Sent Events, large downloads, and WebSockets produce extreme durations at both levels. Exclude or baseline them separately or they will dominate the histograms.
Warm-up distortion. After a restart or reload, backend connection pools are empty and every request pays TCP plus TLS setup to the upstream. Both metrics read high for a few minutes. Do not compute the gap during warm-up.
OTel unit change. If you export via OpenTelemetry rather than Prometheus, note that v3.3.4 standardised the request duration metric unit from milliseconds to seconds. Mixed-version fleets will show a 1000x discrepancy that has nothing to do with middleware.

Signals to watch in production

Signal	Why it matters	Warning sign
Entrypoint p95 minus service p95	The overhead itself	Gap > 2x its own baseline for > 5 minutes
`traefik_entrypoint_request_duration_seconds`	Client-perceived latency at the edge	p95 > 2x baseline while service latency is flat
`traefik_service_request_duration_seconds`	Backend-side latency	Rising in lockstep with entrypoint (backend problem, not Traefik)
`process_cpu_seconds_total` rate	TLS, gzip, and regex all burn CPU	CPU climbing with the gap while request rate is flat
`traefik_entrypoint_requests_tls_total` rate	New TLS sessions drive handshake cost	Spike correlating with gap widening
`go_gc_duration_seconds`	GC pauses inflate every request uniformly	p99 pauses > 100ms alongside a widened gap
`traefik_service_retries_total`	Retries distort the service-side measurement	Retry rate > 5% of request rate

Alert on the gap’s deviation from its own baseline, not on an absolute threshold. A static file service and a compute API have legitimately different overhead profiles, and universal thresholds will either page you on noise or miss real middleware regressions.

How Netdata helps

Netdata collects the Traefik Prometheus histograms per second, so the entrypoint and service duration series are available at a granularity fine enough to see the gap open during an incident rather than after it.
Charting entrypoint p95, service p95, and the derived difference side by side turns the subtraction technique into a standing panel instead of an ad-hoc PromQL query at 3 a.m.
Correlating the gap against process_cpu_seconds_total and the TLS request rate in the same dashboard answers the first follow-up question (handshake-bound vs middleware-bound) without switching tools.
Go runtime metrics (go_goroutines, heap, GC pauses) are collected alongside Traefik’s own metrics, so GC-driven latency inflation shows up on the same timeline.
ML anomaly detection on the gap series flags deviations from the learned baseline, which handles the “universal thresholds do not work” problem described above.