$ guides / traefik / traefik-traffic-drop-to-zero ▌

Operations Guides

Traefik traffic dropped to zero: entrypoint request rate flatlined

The request rate on an entrypoint that normally serves traffic has fallen to zero, or close enough. Clients are timing out or getting connection errors. Sometimes Traefik’s process is still running and /ping still returns 200, which makes this worse: your health checks are green while no traffic is flowing.

There are two fundamentally different families of root cause, and telling them apart early is the whole game. Either traffic is not reaching Traefik at all (DNS, cloud load balancer, firewall, network partition upstream of the proxy), or traffic is arriving at the host but Traefik cannot accept it (a dead listener, file descriptor exhaustion, a port-bind failure after restart). The checks below are ordered to split those two families within the first few minutes.

A note on detection: a flatlined entrypoint is an anomaly-detection problem, not a static-threshold problem. rate(traefik_entrypoint_requests_total[5m]) == 0 will also fire at 3 a.m. on a genuinely quiet service. What makes this an incident is the deviation from the established baseline for that entrypoint at that time of day, sustained for more than a few minutes. If you are reading this because an alert fired, confirm the baseline deviation before assuming an outage.

What this means

traefik_entrypoint_requests_total is a counter with labels {code, method, protocol, entrypoint}. It increments once per HTTP request accepted at the entrypoint listener. If the rate is zero, one of three things is true:

No packets arrive. Something upstream of Traefik (DNS record, cloud LB target health, security group, firewall, upstream router) is sending traffic elsewhere or nowhere. Traefik is fine; the path to it is not.
Packets arrive but nothing accepts them. The entrypoint listener is gone or broken. The process can stay alive and healthy-looking in this state. There is a long-running upstream bug (traefik/traefik#8841, open since 2022 and still reported on recent v3.x releases) where a failed setsockopt on an accepted connection kills the listener’s accept loop, leaving set tcp 10.0.0.1:443: setsockopt: invalid argument in the logs and a dead entrypoint behind a living process.
Traefik is alive but saturated. File descriptor exhaustion is the classic case: at the FD limit, 100% of new connections fail instantly while existing connections keep working. From the outside this looks like intermittent or total connection refusal. From the metrics it looks like a request-rate collapse with traefik_open_connections pinned at a plateau.

The less common direction matters too: if you came here because of a sudden spike rather than a drop, that is a different problem (traffic flood or DDoS), and the diagnostics below do not apply.

flowchart TD
  A[Entrypoint request rate flatlined] --> B{Process alive and port listening?}
  B -->|No| C[Process dead or bind failed]
  B -->|Yes| D{SYNs arriving at the host?}
  D -->|No| E[Upstream problem: DNS, cloud LB, firewall, partition]
  D -->|Yes| F{FD ratio near limit?}
  F -->|Yes| G[FD exhaustion: new connections refused]
  F -->|No| H{setsockopt errors in logs?}
  H -->|Yes| I[Dead accept loop, listener zombie: restart Traefik]
  H -->|No| J[Check open connections, TLS, and recent deploys]

Common causes

Cause	What it looks like	First thing to check
Cloud LB or upstream health check failing	LB targets marked unhealthy, traffic drained; Traefik itself fine	LB target health in the cloud console; TCP connect to the entrypoint from outside the LB
DNS or upstream network partition	Rate fell to zero instantly across all entrypoints at once	Resolve the public hostname, trace the path, check upstream provider status
Firewall or security group change	Connections time out rather than refuse; often follows an infra change	`ss -tn state syn-recv` on the host while a client retries
FD exhaustion	`process_open_fds / process_max_fds` at or near 1; existing connections work, new ones fail	`process_open_fds` vs `process_max_fds` on the metrics endpoint
Listener accept loop dead (setsockopt bug)	Process alive, `/ping` 200, port may still appear listening, but zero accepts; `setsockopt: invalid argument` in logs	`grep setsockopt` on Traefik logs
Port-bind failure after restart	Process restarted recently and never re-bound the entrypoint; “listen tcp :443: bind: address already in use” style errors	`process_start_time_seconds`, then logs from startup
Unexpected graceful shutdown	Logs show “Stopping server gracefully” with no operator action	Traefik logs around the rate drop
Legitimate traffic migration	Rate dropped here but rose on another instance or entrypoint	Compare `traefik_entrypoint_requests_total` across all instances and entrypoints

Quick checks

Run these from the Traefik host or pod. All are read-only. Port 8080 below assumes the default internal entrypoint where /ping and /metrics live; adjust for your configuration. Adjust the log path in check 7 as well (or use docker logs / journalctl -u traefik).

# 1. Is the process alive, and when did it start?
pgrep -af traefik
grep 'Max open files' /proc/$(pgrep -f traefik | head -1)/limits

# 2. Is the entrypoint port actually listening?
ss -tlnp | grep -E ':(80|443)\b'

# 3. Are connection attempts arriving? (SYN received but not answered, or nothing at all)
ss -tn state syn-recv | head
ss -s

# 4. FD usage: current vs limit
ls /proc/$(pgrep -f traefik | head -1)/fd | wc -l

# 5. Metrics: request rate, open connections, FD ratio, reload freshness
curl -s http://localhost:8080/metrics | grep -E 'traefik_entrypoint_requests_total|traefik_open_connections|process_open_fds|process_max_fds|traefik_config_last_reload_success'

# 6. Ping (liveness only - remember /ping says nothing about listeners or routing)
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080/ping

# 7. The silent-listener bug signature and shutdown signatures
grep -iE 'setsockopt|stopping server|graceful' /var/log/traefik/traefik.log | tail -20

# 8. TCP connect from an outside host, bypassing any LB
nc -zv <traefik-host-ip> 443

Interpretation shortcuts:

Port not listening + recent start time: bind failure or a crash-looping process that keeps losing the port race. Check startup logs for the bind error.
Port listening + zero SYNs arriving: the problem is upstream of the host. Stop debugging Traefik.
SYNs arriving + FD ratio near 1: FD exhaustion. New connections cannot be accepted.
Port listening + SYNs arriving + FDs fine + setsockopt in logs: the accept loop is dead. Only a restart recovers the listener.
Everything normal on this instance: check whether another instance took the traffic (failover, DNS change, LB weight change). A flatline plus a matching spike elsewhere is a migration, not an outage.

How to diagnose it

Confirm the anomaly is real. Compare the current rate against the same time-of-day baseline for that entrypoint. Nights, weekends, and planned maintenance windows flatline entrypoints legitimately. Also confirm you are summing correctly: a label change or a new instance name in the query can look like a traffic drop.
Split upstream vs local. From a host outside your network path (or at least outside the LB), attempt a TCP connect to the entrypoint. Simultaneously watch ss -tn state syn-recv on the Traefik host. If nothing arrives, the fault is DNS, the cloud LB (target health, listener rules), a security group, or a network partition. Escalate there; Traefik is a victim, not a cause.
Check the FD ratio. Compute process_open_fds / process_max_fds. Above 0.95, you are at the cliff edge: existing keep-alive connections continue to serve (which is why some traffic may still flow) but new connections are refused instantly. If process_max_fds is 1024, that is the default container limit and it is too low for a production edge proxy.
Look for the dead-listener signature. Search Traefik logs for setsockopt: invalid argument or setsockopt: operation not supported. The first is the long-standing bug in which a failed setsockopt on an accepted connection propagates up and kills the listener’s accept loop while the process keeps running (issue #8841). The second was caused by MPTCP support in v3.4.2 through v3.4.4 and was fixed in v3.4.5 by removing MPTCP; if you are on those versions, upgrade. Whether upgrading past v3.4.5 also resolves the “invalid argument” variant is unconfirmed.
Rule out an unexpected shutdown or restart. Look for “Stopping server gracefully” without an operator action, and check process_start_time_seconds for a recent restart. There are upstream reports of spontaneous graceful stops; in at least one reported case an upgrade resolved it. Also check whether a container healthcheck or orchestrator event (eviction, node drain, OOM kill of a sidecar) triggered it.
Check for provider desync as a contributing factor. Provider disconnection does not normally zero out an entrypoint (stale routes keep serving), but if the rate drop coincided with a mass route change, check traefik_config_last_reload_success age and entrypoint-level 404s. If traffic is arriving but every request gets a Traefik-generated 404, you want Traefik 404 not found: requests arriving with no matching router instead.
If everything local is clean, widen the blast radius check. Compare request rates across all Traefik instances and all entrypoints. Traffic that vanished here and appeared elsewhere is a routing or LB decision upstream. Traffic that vanished everywhere is DNS, a shared LB, or a genuine demand change.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`rate(traefik_entrypoint_requests_total[5m])` by entrypoint	The flatline itself	Sustained >50% drop from the same-time-of-day baseline without a known cause
`process_open_fds / process_max_fds`	FD exhaustion zeroes new connections before any other signal moves	Ratio >0.80 trending up; >0.95 is cliff edge
`traefik_open_connections` by entrypoint	Plateau at the limit while request rate falls means connections cannot be accepted or closed	Flat ceiling coinciding with the rate drop
`process_start_time_seconds`	Catches restarts and crash loops behind a flatline	Recent change matching the rate drop
`traefik_config_last_reload_success` age	Stale config after provider loss can strand routes	Timestamp frozen while deployments are happening
`traefik_entrypoint_requests_total{code="404"}`	Distinguishes “no traffic” from “traffic arriving but no router matched”	Rising 404 share alongside falling total rate
Scrape target up / `/ping`	Baseline liveness; absence of metrics is itself a signal	Target down, or `/ping` non-200

Fixes

Upstream reachability (DNS, LB, firewall)

Restore the path, not the proxy. Traefik is healthy in this scenario; restarting it accomplishes nothing. Re-point or repair DNS, fix the LB health check or target group, or roll back the firewall/security-group change. The common trap: LB health checks that target an entrypoint port Traefik stopped serving, or a timeout mismatch between the cloud LB idle timeout and Traefik’s respondingTimeouts.idleTimeout causing the LB to reuse dead connections. Align those timeouts so the LB’s idle timeout is shorter than Traefik’s.

FD exhaustion

Raise the limit and restart to clear the pressure. The immediate relief is a restart, which closes all FDs, but it drops every in-flight connection, so treat it as a deliberate action, not a reflex. Then fix the limit: 1024 (the common container default) is not viable for an edge proxy. Set a high nofile ulimit in the container spec, compose file, or systemd unit. Afterward, investigate why FDs grew: long-lived WebSocket/gRPC connections, a connection leak (traefik_open_connections growing without matching request-rate growth), or keep-alive misconfiguration.

Dead accept loop (setsockopt bug)

Restart is the only reliable recovery. There is no confirmed fix upstream as of this writing; restart Traefik and the listener recovers. Because the process stays alive and /ping stays green, add detection that survives a “healthy” process: alert on the entrypoint rate anomaly itself, and grep logs for setsockopt as a confirmation signal. If you are on v3.4.2 through v3.4.4, upgrade to v3.4.5 or later to eliminate the MPTCP variant.

Bind failure after restart

Find what owns the port. Check startup logs for the bind error, then ss -tlnp to see what holds the port. Common cases: a previous Traefik process not fully terminated, two replicas scheduled to the same host port, or missing capabilities in a hardened container. Fix the conflict and let the process rebind cleanly.

Unexpected graceful shutdown

Identify the trigger before restarting blindly. Check orchestrator events (evictions, node drains, healthcheck failures) and any external process supervisor. If logs show a graceful stop with no external signal, note the version and consider upgrading, since at least one reported instance of this pattern was resolved by an upgrade.

Prevention

Alert on baseline deviation, not zero. Static rate == 0 alerts false-fire on quiet services. Alert on a sustained drop versus the same-time-of-day baseline, combined with a traffic floor where one exists. Corroborate with traefik_open_connections and the FD ratio so a single metric cannot page you alone.
Monitor the FD ratio with headroom. Page at >0.95 sustained and rising; ticket at >0.80. Keep steady state below 70% of process_max_fds.
Raise FD limits deliberately. Never run a production Traefik at the 1024 default.
Log-scan for setsockopt. It is the earliest reliable indicator of the dead-listener bug; the process will not tell you otherwise.
Do not trust /ping. It proves the process is alive, nothing more. It does not check listeners, routing, providers, or certificates.
Watch per-instance rates in HA. A flatline on one replica with healthy siblings is an instance-level fault; a flatline everywhere is upstream. Compare traefik_config_last_reload_success across replicas to catch drift.

How Netdata helps

Per-second entrypoint throughput makes the exact moment of the flatline visible, which lets you align it with deploys, orchestrator events, and log lines instead of guessing at the timeline.
FD ratio tracking (process_open_fds vs process_max_fds) alongside open connections shows the exhaustion cliff forming before the request rate collapses, turning a page into a ticket.
Process restart detection via start-time changes correlates a traffic drop with a crash loop or bind failure without manual log archaeology.
Anomaly detection on request rates handles the “what is normal varies per deployment” problem: it flags deviations from learned baselines per entrypoint rather than relying on static thresholds.
Cross-signal correlation in one view (request rate, open connections, FDs, config reload freshness, 404 share) is what separates the six causes in the table above quickly, instead of checking each in isolation.