Traefik file descriptor monitoring: process_open_fds, limits, and headroom

Traefik file descriptor exhaustion is a cliff-edge failure. The proxy works until it hits its OS limit on open files, then 100% of new connections fail instantly with “too many open files” errors. Existing connections keep working, so dashboards can look calm while every new client is dropped. There is no graceful degradation.

The most obvious Traefik connection metric, traefik_open_connections, does not measure the thing that kills you. It tracks only entrypoint connections, a subset of total FD usage. The comprehensive signal is the ratio process_open_fds / process_max_fds from the Go process collector, which counts everything the process holds open: client sockets, backend sockets, provider connections, log files, ACME storage, and pipes.

This article covers how to read those metrics, where the FDs go, how to set thresholds that page at the right time, and how to size limits and headroom for production.

What this means

Every proxied HTTP/1.1 connection consumes two file descriptors: one for the client-facing socket accepted at the entrypoint, and one for the backend-side socket from the connection pool. HTTP/2 can multiplex many streams over one backend connection, so backend FD use is protocol-dependent. On top of that, the process holds FDs for provider connections (Docker socket, Kubernetes API watches, Consul, file provider watches), access log files, and ACME certificate storage.

The process FD limit comes from the OS: whatever ulimit the process started with, read from /proc/<pid>/limits and surfaced as process_max_fds. A 1024 soft limit can survive from a low host or daemon configuration. Docker does not impose a container nofile default; without daemon default-ulimits, the container inherits the daemon limit. Modern systemd/containerd hosts often use a much larger limit such as 524288 or 1048576, so verify the effective value rather than assuming 1024. A single browser session can hold half a dozen connections; a modest production workload exhausts 1024 FDs in seconds.

flowchart TD
  L[process_max_fds - OS limit]
  L --> C[Client connections - 1 FD each]
  L --> B[Backend connections - 1 FD each]
  L --> P[Provider watches - Docker, K8s API, files]
  L --> F[Log files and ACME storage]
  C --> U[process_open_fds - total in use]
  B --> U
  P --> U
  F --> U
  U --> R{Ratio open / max}
  R -->|below 70%| OK[Healthy steady state]
  R -->|above 80%| T[TICKET - eroding headroom]
  R -->|above 95% and rising| PG[PAGE - exhaustion imminent]

Note that traefik_open_connections (labels: entrypoint, protocol in v3) maps only to the client connections box. It will never warn you about the other three consumers, and it can look flat while FD usage climbs from a backend connection leak.

Common causes

Cause	What it looks like	First thing to check
FD limit too low (1024 can survive from host/daemon config)	`process_max_fds` is 1024; failures start under moderate load	`grep 'Max open files' /proc/<pid>/limits`
WebSocket/gRPC baseline	High but stable `process_open_fds`; long-lived connections inflate the floor	`traefik_open_connections` trend vs request rate
Backend connection leak	FDs rising while `traefik_open_connections` is flat	`ss -tnp` socket states for the Traefik PID
Traffic spike / DDoS	FDs and entrypoint connections rising together	`traefik_entrypoint_requests_total` rate
Log file or inotify FD leak	FDs rising with no matching connection growth	Count FDs by type in `/proc/<pid>/fd`

Quick checks

# Count open FDs right now
ls /proc/$(pgrep traefik)/fd | wc -l

# Read the actual limit (soft limit is what process_max_fds reports)
grep 'Max open files' /proc/$(pgrep traefik)/limits

# Pull both gauges from the metrics endpoint
curl -s http://localhost:8080/metrics | grep -E 'process_open_fds|process_max_fds'

# Entrypoint connections (the subset, for comparison)
curl -s http://localhost:8080/metrics | grep traefik_open_connections

# Socket states held by the Traefik process
ss -tnp | grep traefik | awk '{print $2}' | sort | uniq -c

Three readings to internalize: if process_max_fds is 1024, that is your root cause and nothing else matters until you fix it. If CLOSE_WAIT sockets are accumulating, backends closed connections that Traefik never cleaned up. If TIME_WAIT is exploding, connection pooling to backends is ineffective and you are burning FDs and ephemeral ports on churn.

How to diagnose it

Establish the ratio. Compute process_open_fds / process_max_fds. Below 70% is healthy steady state. Between 70% and 80% you are in planning territory. Above 80% you should be actively working on it.
Determine the trend. A single reading tells you position, not direction. Sample the ratio over 10 to 15 minutes. Rising means a leak or growing load. Flat and high means a legitimate baseline (usually long-lived connections) that needs a bigger limit, not a leak hunt.
Split connection FDs from other FDs. Compare traefik_open_connections (summed across entrypoints) against process_open_fds. The gap is backend connections plus provider watches, logs, and storage. If the gap grows while entrypoint connections are flat, the leak is on the backend side or in non-connection FDs.
Correlate with traffic. If traefik_open_connections and process_open_fds rise together with traefik_entrypoint_requests_total, it is real load: you need capacity, not debugging. If connections grow without request growth, something is holding connections open: slow backends, missing timeouts, or a leak.
Check the limit configuration. For containers, verify the ulimit was actually applied (--ulimit nofile=... on docker run, or the ulimits section in Compose). For systemd units, check LimitNOFILE in the [Service] section. Then confirm with /proc/<pid>/limits, because the runtime value is the only one that matters.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`process_open_fds / process_max_fds`	The comprehensive saturation ratio; the only correct FD alert basis	Above 80%, or rising steadily toward it
`process_max_fds`	Reveals a dangerously low limit at a glance	1024 (a low inherited or explicitly configured limit that survived to production)
`traefik_open_connections`	Entrypoint connection subset; separates client load from other FD consumers	Growing without a matching request rate increase
`traefik_entrypoint_requests_total`	Tells you whether connection growth is real traffic	Rate flat while connections climb
CLOSE_WAIT / TIME_WAIT counts (OS)	Connection cleanup health between Traefik and backends	CLOSE_WAIT growing over time; TIME_WAIT in the tens of thousands

Alert thresholds that match the failure shape:

TICKET at 80%. Headroom is eroding. This is a business-hours signal to plan capacity, raise the limit, or find the leak. The 80% level exists because FD usage is normally far below the limit, so reaching it means something has already changed.
PAGE at 95%, sustained for more than 2 minutes AND still rising or not declining. The sustained-and-rising condition is what makes this pageable. A deployment intentionally running hot at 95% with a flat trend is a capacity decision, not a 3 a.m. event. But 95% and climbing means exhaustion is minutes away regardless of cause, and the failure is total the moment it arrives.

Fixes

Raise the limit

If process_max_fds is 1024 or another low default, raising it is the fix, full stop. Production Traefik should run with at least 65536. For Docker, pass --ulimit nofile=65536:65536 or set ulimits in the Compose file. For systemd, set LimitNOFILE in the unit and reload the daemon. Both require a process restart to take effect, so schedule it. Verify afterward against /proc/<pid>/limits, not against the config file you edited.

Right-size the baseline

If long-lived WebSocket or gRPC traffic legitimately inflates FD usage, do not tune alerts to silence the noise. Baseline those workloads separately, set the limit so observed steady state sits below 70%, and keep the 30% headroom for connection bursts: HTTP/1.1 bursts, WebSocket reconnect storms after a network blip, mass client reconnects after a deploy.

Fix backend connection hygiene

If the leak is on the backend side (rising FDs, rising CLOSE_WAIT, flat client traffic), the proxy is holding connections backends already abandoned. Review backend transport timeouts and idle connection settings, and confirm backends are not sending Connection: close on every response, which defeats pooling and churns FDs.

Emergency mitigation during an active incident

If you are paged at 95% and rising: restart the Traefik process or pod to release leaked FDs and buy time. This drops all in-flight connections, so treat it as the disruptive action it is, then immediately raise the limit and start the leak investigation above. A restart without a limit change or a root cause is a recurring incident.

Prevention

Verify the limit in every deployment path. Container specs, Compose files, host daemon limits, and systemd units each have their own way to leave or revert the soft nofile limit at 1024. Check process_max_fds itself in your dashboards; it is a metric, so alert or at least report on it being low.
Keep steady state below 70%. The 30% headroom absorbs spikes. If your normal baseline is above 70%, raise the limit rather than accepting the risk.
Alert on the ratio, never on traefik_open_connections alone. The entrypoint gauge is a diagnostic signal for capacity planning and leak detection, not an exhaustion alarm.
Estimate runway during growth. If FD usage grows linearly, time to exhaustion is (process_max_fds - process_open_fds) / rate_of_growth. Watch the derivative during incidents so you know whether you have minutes or hours.
Baseline WebSocket and gRPC workloads separately. Their long-lived connections set the floor; alerting tuned to HTTP traffic patterns will misfire in both directions.

How Netdata helps

Netdata charts process_open_fds and process_max_fds per process at per-second resolution, so the ratio and its trend are visible without hand-rolled PromQL.
Correlating FD usage with traefik_open_connections and entrypoint request rate on one dashboard separates real load from a leak in seconds instead of grep sessions.
Per-second collection catches the steep final climb of the FD cliff that minute-resolution scrapes smooth over: the difference between a 2-minute warning and no warning.
Socket state and per-process resource charts alongside Traefik metrics surface CLOSE_WAIT and TIME_WAIT buildup before the FD ratio moves.
Anomaly detection on process_open_fds flags growth that deviates from the established baseline, which is exactly the leak signature static thresholds miss.