Traefik request rate monitoring: entrypoint and service throughput baselines

Traefik exposes two request counters that look similar but measure different things. traefik_entrypoint_requests_total counts every request that arrives at a listener, whether or not Traefik finds a route for it. traefik_service_requests_total counts only requests that matched a router, survived the middleware chain, and were proxied to a backend. The difference between the two is where most routing and middleware problems show up first.

Static thresholds (“page if requests/sec drops below 50”) fail on a reverse proxy fronting heterogeneous workloads. A static asset service and a compute-heavy API differ by orders of magnitude, and the same entrypoint differs by time of day. This article covers what each counter counts, how to read the gap between them, and how to build per-entrypoint, per-service, per-time-of-day baselines that catch real faults without paging on legitimate traffic dips.

What each counter counts

traefik_entrypoint_requests_total is a counter with labels code, method, protocol, and entrypoint. Every request that completes at a listener increments this series, including requests that matched no router at all. When no router matches, Traefik itself returns a 404 at the edge, and that response shows up here as code="404" with no corresponding increment anywhere in the service metrics, because no service was ever selected.

In current v3 releases, Prometheus addEntryPointsLabels defaults to true; older v2 releases defaulted it to false. traefik_service_requests_total is a counter with labels code, method, protocol, and service. It increments when a request is proxied to a backend service. The code label on both families is what turns a raw throughput number into an error ratio: 5xx at the service level means the backend (or Traefik’s attempt to reach it) failed, while 404 at the entrypoint level means Traefik’s routing table had no answer.

A third family, traefik_router_requests_total, sits between the two, but per-router metrics are off by default. Enabling them requires addRoutersLabels: true, which carries a real cardinality cost on deployments with many routes. For most baseline work, the entrypoint and service families are sufficient.

These are labeled Prometheus counter series, so a label combination only exists after the first matching request. A newly deployed service shows nothing until traffic actually flows through it, and a service with zero traffic is indistinguishable from a service with no route unless you also check the entrypoint side.

How a request moves through the counters

The two counters bracket Traefik’s processing pipeline. Knowing where requests drop out between them is the whole diagnostic model.

flowchart LR
  client[Client] --> ep[Entrypoint
entrypoint_requests_total++]
  lbprobe[Cloud LB health probe] --> ep
  ep --> router{Router match?}
  router -- "no match" --> edge404[Traefik returns 404
no service counter]
  router -- "match" --> mw{Middleware chain}
  mw -- "short-circuit: auth, rate limit" --> mwreject[Rejected at edge
no service counter]
  mw -- "pass" --> svc[Service / load balancer
service_requests_total++]
  svc --> backend[Backend]

Every request increments the entrypoint counter. Only requests that match a router and pass the full middleware chain increment the service counter. Entrypoint counts are therefore always greater than or equal to service counts, and the gap between them is not noise. It is composed of:

Edge 404s. Requests for hostnames or paths with no matching router. Traefik returns these itself; no backend is involved.
Middleware short-circuits. Authentication rejections, rate limiter responses, circuit breaker responses. Any middleware can terminate the chain before a service is selected, and there are no per-middleware Prometheus metrics. The only instrumentation points are entrypoint, router, and service.
Synthetic traffic. Health probes from an upstream cloud load balancer hit the entrypoint and are counted like any other request.

Reading the gap

The most useful single pattern in Traefik throughput monitoring is this: entrypoint rate steady, service rate dropping means routes are being lost.

Traffic is still arriving at the listener, but a shrinking fraction of it reaches a backend. The usual suspects are middleware rejections (an auth middleware suddenly denying everything, a rate limiter tripping) or router misconfiguration (a bad annotation or label that silently prevents route creation). Traefik ignores unknown annotations and misspelled label keys without an error metric, so a deployment can remove a route without anything firing. Cross-check with traefik_entrypoint_requests_total{code="404"} to separate “no router matched” from “middleware rejected,” and with the Traefik API (/api/http/routers) to see what is actually loaded versus what you intended.

The reverse pattern, entrypoint rate dropping while service rate holds, is rarer and usually means one entrypoint lost traffic upstream (DNS, cloud LB, firewall) while another continues to serve.

Two distortions to correct for before trusting either number:

Cloud LB health probes inflate the entrypoint rate. If Traefik sits behind a cloud load balancer, the LB’s health checks arrive at the entrypoint and increment traefik_entrypoint_requests_total like client traffic. At low real traffic volumes, probe traffic can dominate the series and mask a genuine drop. Traefik’s /ping endpoint is typically served on its own entrypoint (port 8080 in most deployments), so point LB health checks there rather than at a serving entrypoint. Probes that do hit a serving entrypoint are indistinguishable from real requests in the counter.

Retry middleware inflates the service rate. When retries are enabled, each attempt increments the service counter, so traefik_service_requests_total can exceed the actual client request count. A rising service rate is not always rising demand; correlate with the retry middleware configuration and access-log RetryAttempts before concluding anything. Under retry load the client-perceived request rate is lower than the service-level rate.

Building baselines that work

The playbook-level approach is to establish hourly baselines per entrypoint and alert on deviation from the same time of day, not from a global average:

Collect the raw counters. Scrape the metrics endpoint and confirm both families are present with the labels you expect:

# Inspect current counters (read-only)
curl -s http://localhost:8080/metrics | grep traefik_entrypoint_requests_total
curl -s http://localhost:8080/metrics | grep traefik_service_requests_total

For a manual rate estimate, take two samples 60 seconds apart, compute the delta, and divide by 60.

Split by entrypoint and service, never aggregate across them. A single “total requests” line hides a dead service behind a busy one. In Kubernetes, service label values follow the <namespace>-<ingressroute-name>-<hash> pattern, which is hard to read; build a mapping to real service names once and keep it in the dashboard.
Baseline per time of day. Compare 14:00 today against 14:00 on recent comparable days, not against the 24-hour average. Nights and weekends are legitimate low-traffic windows and will false-fire any flat threshold.
Alert on deviation, not level. A sustained drop of more than 50% from the same-time-of-day rolling average, lasting more than 5 minutes without a known deployment cause, is a reasonable ticket threshold. A drop to zero on a normally active entrypoint during business hours is the one case that justifies a page.
Baseline the gap, not just the rates. Track the ratio of service-level to entrypoint-level requests per time window. A shift in that ratio is often the earliest sign of route loss or a middleware behaving differently, and it is largely immune to organic traffic swings because both sides move together.
Separate the error ratio from the volume. Use the code label to compute 5xx and 4xx ratios alongside the rate. A service holding its request rate while its 5xx ratio climbs is a backend problem, not a routing problem. Entrypoint-level 404s (no router matched) and service-level 404s (the backend returned 404) have completely different investigation paths; keep them as separate series.

A caveat on the gap math: service metrics carry no entrypoint label, so you cannot compute a clean per-entrypoint-versus-per-service difference when one entrypoint fronts many services. Compute the gap at the aggregate level, then drill into code="404" per entrypoint and per-service rates to localize it.

Signals to watch in production

Signal	Why it matters	Warning sign
`rate(traefik_entrypoint_requests_total[5m])` per entrypoint	Fundamental edge throughput; a drop means traffic is not arriving or Traefik cannot accept it	Sustained >50% deviation from same-time-of-day baseline; zero on an active entrypoint
`rate(traefik_service_requests_total[5m])` per service	Whether traffic actually reaches backends	Dropping while entrypoint rate is steady (routes lost); zero on a normally busy service
Entrypoint `code="404"` rate	Requests matching no router; routing table correctness	Sustained >5% of entrypoint requests, or >3x baseline, especially after a deploy
Service 5xx ratio from `code` label	Backend health from Traefik’s perspective	>1% sustained 5 minutes (ticket); >5% sustained 2 minutes (page)
Service-to-entrypoint rate ratio	Detects route loss and middleware behavior shifts independent of volume	Ratio shift without a known config or middleware change
Access-log `RetryAttempts` and retry middleware config vs service rate	Distinguish real demand from retry amplification; the Prometheus retry counter is not wired in current v2/v3	Retry attempts >5% of service requests sustained; attempt count approaching request count

How Netdata helps

Netdata charts traefik_entrypoint_requests_total and traefik_service_requests_total as rates split by code, entrypoint, and service, so the steady-edge/dropping-service pattern is visible on one dashboard without hand-built queries.
Per-second collection granularity catches short route-loss windows and probe-driven distortions that 60-second scrape intervals smooth over.
Splitting request volume by response code keeps entrypoint 404s (routing problems) visually separate from service 404s (application problems).
Plotting service request rate next to retry rate makes retry inflation of the throughput numbers obvious.
Historical retention per series supports the same-time-of-day baseline comparisons this article recommends, instead of alerting against flat thresholds.
Per-metric anomaly detection flags deviation from learned normal behavior per entrypoint and service, which maps directly onto the baseline-deviation alerting model above.