Traefik too many open files: file descriptor exhaustion at the edge

Your Traefik logs are filling with accept4: too many open files errors. New clients cannot connect. Requests that do get through may return 502 or 503. Meanwhile, some clients insist everything works fine, because their existing connections are still alive.

This is file descriptor (FD) exhaustion. Traefik sits at the edge and holds at least two FDs for each proxied HTTP/1.1 connection: one for the client side, one for the backend side. HTTP/2 can multiplex streams over one backend connection, so backend FD use is protocol-dependent. Add provider connections (Docker socket, Kubernetes API watches), log files, and ACME storage handles, and the count climbs fast. When it hits the OS limit, every new accept4() call fails instantly.

The failure is a cliff edge. There is no graceful degradation: below the limit everything works, at the limit 100% of new connections fail while existing connections keep serving traffic. That asymmetry is why this incident confuses people: health checks riding existing connections pass, /ping returns 200, and yet real users are locked out.

What this means

Every TCP connection, open file, and pipe a process holds consumes one file descriptor. Linux enforces a per-process cap (Max open files in /proc/<pid>/limits), and when a process reaches it, the kernel refuses new opens. For a proxy, that means new client connections are refused at the accept loop.

The error surfaces in Traefik’s logs as an accept error on the affected entrypoint. Since Traefik v2.0, the accept loop retries temporary accept errors with backoff instead of killing the listener, so the process survives but keeps logging the failure and refusing new connections until FDs are freed or the process is restarted.

The single most common root cause is not a leak. It is a low inherited or misconfigured FD limit; 1024 is far too low for an edge proxy. Docker has no built-in container nofile default and inherits the daemon limit when default-ulimits is unset, while modern containerd/systemd hosts often use much larger values. A single browser session can hold six connections; a load balancer health-checking with keep-alive disabled burns one connection per probe. At any real traffic level, 1024 FDs is minutes or hours of runway, not a capacity plan.

flowchart TD
  A[New client connection arrives] --> B{FD available?}
  B -->|yes| C[Connection accepted, proxied normally]
  B -->|no: process at Max open files| D[accept4 fails: too many open files]
  D --> E[New client dropped instantly]
  F[Existing connections] --> G[Keep working, already hold FDs]
  H[FD consumers] --> I[Client connections]
  H --> J[Backend connections]
  H --> K[Provider sockets, logs, ACME storage]

Common causes

Quick checks

All read-only and safe to run during an incident.

# 1. Find the Traefik PID
pgrep -f traefik

# 2. Check the effective FD limit for the process
grep 'Max open files' /proc/$(pgrep -f traefik)/limits

# 3. Count currently open FDs
ls /proc/$(pgrep -f traefik)/fd | wc -l

# 4. Check FD usage via metrics (v2/v3, Prometheus endpoint)
curl -s http://localhost:8080/metrics | grep -E 'process_open_fds|process_max_fds'

# 5. Look for the accept errors in the logs
journalctl -u traefik --since "10 min ago" | grep -i "too many open files"
# or, in a container:
docker logs traefik --since 10m 2>&1 | grep -i "too many open files"

# 6. See socket states held by Traefik
ss -tnp | grep traefik | awk '{print $1}' | sort | uniq -c

# 7. Confirm Traefik still thinks it is healthy (it will)
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080/ping

Note on check 7: /ping returning 200 tells you nothing about FD state. It only proves the process is alive. This is the /ping trap: Traefik can report healthy while refusing every new connection.

How to diagnose it

1. Confirm the limit. If /proc/<pid>/limits shows Max open files at 1024, you have likely found the root cause already. A production edge proxy at 1024 will hit this wall; the only question was when.
2. Confirm saturation. Divide current FD count by the limit. At or near 100% with accept4 errors in the logs, this is the incident. Below 70%, the errors may be transient or from a different process; keep looking.
3. Determine growth shape. Scrape process_open_fds twice, 60 seconds apart. Flat and high means steady-state overload: the limit is too small for the workload. Rising steadily without a matching request rate increase means a leak.
4. Correlate FDs with connections. Compare traefik_open_connections against the FD count. FDs tracking connections roughly 2:1 is normal proxy behavior. FDs rising while connections stay flat points at non-connection consumers: log files held open after rotation, provider sockets, or backend connections stuck in CLOSE_WAIT.
5. Check socket states. A large CLOSE_WAIT count from the ss output in check 6 means backends closed connections that Traefik never cleaned up: a leak pattern. A large ESTABLISHED count matching the connection metric means genuine load, not a leak.
6. Check goroutines. Rising go_goroutines alongside rising FDs and flat request rate points at hung backend connections: a backend accepted the TCP connection but never responds, and the goroutine waits with the FD held.
7. Check the timing. If exhaustion correlates with a deployment, config change, or traffic event, that narrows the cause. FD counts can also spike extremely fast; production reports exist of counts jumping by orders of magnitude in under a minute with no obvious trigger, so do not rule out exhaustion just because the onset was sudden.

Metrics and signals to monitor

Two things to get right. First, alert on the ratio process_open_fds / process_max_fds, not on traefik_open_connections alone. The connection metric is a subset of FD usage and will miss log files, provider sockets, and leaked backend sockets. Second, the page condition should be ratio > 95% sustained for more than 2 minutes AND still rising or not declining. A deployment that intentionally sits at 95% with stable WebSocket connections should not page; the trend condition filters that out.

Fixes

Raise the limit (the real fix for most incidents)

Use 65536 or higher for production. Where you set it depends on how Traefik runs:

# systemd unit (host-installed Traefik): systemd ignores /etc/security/limits.conf
# In the [Service] section of the unit file:
LimitNOFILE=65536

# Then reload and restart
systemctl daemon-reload
systemctl restart traefik

# Docker
docker run --ulimit nofile=65536:65536 ...

# docker-compose.yml
services:
  traefik:
    ulimits:
      nofile:
        soft: 65536
        hard: 65536

For Kubernetes, do not rely on the pod spec to set nofile. Verify /proc/<pid>/limits; raise the CRI/runtime or node service limit where supported, bake an image-level default, or use an init/entrypoint wrapper only if its hard limit permits the raise. Treat the observed process limit as the only authoritative value.

Verify after every change:

# Confirm the new limit took effect
grep 'Max open files' /proc/$(pgrep -f traefik)/limits

Tradeoff: raising the limit buys capacity but does not fix a leak. If FDs were leaking before, they will leak to the new ceiling, just more slowly. Pair the limit increase with ratio-based alerting so you see the climb.

Restart to clear leaked FDs (buys time, not a fix)

Restarting Traefik closes every FD it holds. If the exhaustion was caused by a leak, this immediately restores service.

# Disruptive: drops all in-flight connections
systemctl restart traefik
# or
docker restart traefik
# or, in Kubernetes
kubectl rollout restart deployment/traefik -n <namespace>

Be explicit about what this costs: all existing connections drop. With graceful shutdown configured, in-flight requests drain first, but WebSocket and other long-lived connections still terminate. Do this during an active incident to recover, then schedule the leak investigation as follow-up work. If you restart without investigating, you are on a countdown to the next identical incident.

Fix the leak (if diagnosis showed one)

• Hung backends holding connections: configure serversTransport.forwardingTimeouts.dialTimeout, .responseHeaderTimeout, and .idleConnTimeout so Traefik gives up on backends that accept but never respond.
• CLOSE_WAIT accumulation: backends are closing connections Traefik does not reap; check for intermediaries (firewalls, NAT gateways) silently killing idle connections, and align keep-alive timeouts so Traefik’s idle timeout is shorter than any intermediary’s.
• Long-lived connection workloads: WebSocket and gRPC connections each pin an FD indefinitely. Baseline them separately and size the FD limit for the peak concurrent connection count, not the request rate.

Prevention

• Set the FD limit explicitly, everywhere. Never let a production Traefik inherit a default. 1024 in a container is a scheduled outage. 65536 is the floor, not the ceiling.
• Alert on the ratio with trend. Page at process_open_fds / process_max_fds > 95% sustained 2 minutes and rising; ticket at 80%.
• Keep steady state below 70%. The 30% headroom absorbs connection storms: HTTP/1.1 bursts, WebSocket reconnect waves after a network blip, load balancer failovers.
• Baseline connection count against request rate. A rising traefik_open_connections with flat request rate is your earliest leak signal, weeks before the ratio pages.
• Verify limits after every deployment change. Container image swaps, runtime upgrades, and orchestrator changes can silently reset ulimits. The /proc/<pid>/limits check takes two seconds; automate it.
• Size FDs from protocol concurrency plus overhead. HTTP/1.1 uses client plus backend descriptors; HTTP/2 can multiplex streams over fewer backend descriptors. Add provider sockets, logs, and ACME storage. If you expect 20k concurrent HTTP/1.1 connections, 65536 is barely adequate.

How Netdata helps

• Netdata charts process_open_fds against process_max_fds per process, so the saturation ratio and its trend are visible without writing PromQL during an incident.
• Per-second granularity catches the fast-spike pattern, where FD counts jump in under a minute, that per-minute scraping can smooth away.
• Correlating FD usage with traefik_open_connections and entrypoint request rate on one dashboard separates “genuine load” from “leak”: FDs and connections rising together versus FDs rising alone.
• Goroutine count (go_goroutines) alongside FD count surfaces hung-backend leaks before the ratio reaches paging territory.
• Anomaly detection on the FD ratio flags the slow, weeks-long climb that static thresholds miss until the day it pages.
• Post-restart, the same charts confirm whether the restart actually dropped FD usage to baseline or the leak is already rebuilding.

Related guides

• Traefik 404 not found: requests arriving with no matching router
• Traefik 502 Bad Gateway: when the backend is unreachable or returns garbage
• Traefik 503 Service Unavailable: no healthy backends left in the pool
• Traefik 504 Gateway Timeout: the backend is alive but too slow
• Traefik 5xx error rate: telling Traefik-generated errors from backend errors
• Traefik cascading backend failure: how a partial outage becomes a total one
• Traefik config last reload success: monitoring configuration freshness
• Traefik dashboard returns 404: reaching the API and dashboard correctly
• Traefik health checks pass but requests fail: when the probe lies
• How Traefik actually works in production: a mental model for operators
• Traefik monitoring checklist: the signals every production edge router needs
• Traefik monitoring maturity model: from survival to expert