Your Traefik logs are filling with accept4: too many open files errors. New clients cannot connect. Requests that do get through may return 502 or 503. Meanwhile, some clients insist everything works fine, because their existing connections are still alive.

This is file descriptor (FD) exhaustion. Traefik sits at the edge and holds at least two FDs per proxied connection: one for the client side, one for the backend side. Add provider connections (Docker socket, Kubernetes API watches), log files, and ACME storage handles, and the count climbs fast. When it hits the OS limit, every new accept4() call fails instantly.

The failure is a cliff edge. There is no graceful degradation: below the limit everything works, at the limit 100% of new connections fail while existing connections keep serving traffic. That asymmetry is why this incident confuses people: health checks riding existing connections pass, /ping returns 200, and yet real users are locked out.

What this means

Every TCP connection, open file, and pipe a process holds consumes one file descriptor. Linux enforces a per-process cap (Max open files in /proc/<pid>/limits), and when a process reaches it, the kernel refuses new opens. For a proxy, that means new client connections are refused at the accept loop.

The error surfaces in Traefik’s logs as an accept error on the affected entrypoint. Since Traefik v2.0, the accept loop retries temporary accept errors with backoff instead of killing the listener, so the process survives but keeps logging the failure and refusing new connections until FDs are freed or the process is restarted.

The single most common root cause is not a leak. It is the default container FD limit of 1024, which is far too low for an edge proxy. A single browser session can hold six connections; a load balancer health-checking with keep-alive disabled burns one connection per probe. At any real traffic level, 1024 FDs is minutes or hours of runway, not a capacity plan.

flowchart TD
  A[New client connection arrives] --> B{FD available?}
  B -->|yes| C[Connection accepted, proxied normally]
  B -->|no: process at Max open files| D[accept4 fails: too many open files]
  D --> E[New client dropped instantly]
  F[Existing connections] --> G[Keep working, already hold FDs]
  H[FD consumers] --> I[Client connections]
  H --> J[Backend connections]
  H --> K[Provider sockets, logs, ACME storage]

Common causes

CauseWhat it looks likeFirst thing to check
Default container FD limit (1024)FD count plateaus near 1024 under moderate load; errors start at predictable traffic levelsgrep 'Max open files' /proc/<pid>/limits
Long-lived connections (WebSocket, gRPC)Steady FD growth tracking connection count, not request rateCompare traefik_open_connections trend with traefik_entrypoint_requests_total rate
Backend connection leakFDs rise while open connections stay flat; CLOSE_WAIT sockets accumulatess -tnp filtered to the Traefik PID, count by state
Slow or hung backendsConnections pile up because backends never respond; goroutines grow with FDsgo_goroutines trending up without a traffic increase
Traffic spike or DDoSSudden FD jump correlating with a request rate spiketraefik_entrypoint_requests_total rate at incident onset
Log file rotation failureOld log files held open after rotationls -l /proc/<pid>/fd looking for deleted files still held

Quick checks

All read-only and safe to run during an incident.

# 1. Find the Traefik PID
pgrep -f traefik

# 2. Check the effective FD limit for the process
grep 'Max open files' /proc/$(pgrep -f traefik)/limits

# 3. Count currently open FDs
ls /proc/$(pgrep -f traefik)/fd | wc -l

# 4. Check FD usage via metrics (v2/v3, Prometheus endpoint)
curl -s http://localhost:8080/metrics | grep -E 'process_open_fds|process_max_fds'

# 5. Look for the accept errors in the logs
journalctl -u traefik --since "10 min ago" | grep -i "too many open files"
# or, in a container:
docker logs traefik --since 10m 2>&1 | grep -i "too many open files"

# 6. See socket states held by Traefik
ss -tnp | grep traefik | awk '{print $1}' | sort | uniq -c

# 7. Confirm Traefik still thinks it is healthy (it will)
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080/ping

Note on check 7: /ping returning 200 tells you nothing about FD state. It only proves the process is alive. This is the /ping trap: Traefik can report healthy while refusing every new connection.

How to diagnose it

  1. Confirm the limit. If /proc/<pid>/limits shows Max open files at 1024, you have likely found the root cause already. A production edge proxy at 1024 will hit this wall; the only question was when.
  2. Confirm saturation. Divide current FD count by the limit. At or near 100% with accept4 errors in the logs, this is the incident. Below 70%, the errors may be transient or from a different process; keep looking.
  3. Determine growth shape. Scrape process_open_fds twice, 60 seconds apart. Flat and high means steady-state overload: the limit is too small for the workload. Rising steadily without a matching request rate increase means a leak.
  4. Correlate FDs with connections. Compare traefik_open_connections against the FD count. FDs tracking connections roughly 2:1 is normal proxy behavior. FDs rising while connections stay flat points at non-connection consumers: log files held open after rotation, provider sockets, or backend connections stuck in CLOSE_WAIT.
  5. Check socket states. A large CLOSE_WAIT count from the ss output in check 6 means backends closed connections that Traefik never cleaned up: a leak pattern. A large ESTABLISHED count matching the connection metric means genuine load, not a leak.
  6. Check goroutines. Rising go_goroutines alongside rising FDs and flat request rate points at hung backend connections: a backend accepted the TCP connection but never responds, and the goroutine waits with the FD held.
  7. Check the timing. If exhaustion correlates with a deployment, config change, or traffic event, that narrows the cause. FD counts can also spike extremely fast; production reports exist of counts jumping by orders of magnitude in under a minute with no obvious trigger, so do not rule out exhaustion just because the onset was sudden.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
process_open_fds / process_max_fdsThe only complete FD saturation view; covers connections, files, and socketsRatio > 80% (headroom eroding); > 95% and rising is page-worthy
traefik_open_connectionsEntrypoint connection count; subset of total FD usagePlateau while request rate drops: accept loop is refusing
traefik_entrypoint_requests_total rateDrop to near-zero on a normally busy entrypoint means accepts are failingSustained > 50% drop from baseline without a deployment cause
go_goroutinesOne goroutine per active connection; leaks show here first> 2-3x baseline without a traffic increase
traefik_service_requests_total{code=~"5.."}FD exhaustion at the backend-dial stage surfaces as 502/5035xx spike concurrent with FD ratio climbing

Two things to get right. First, alert on the ratio process_open_fds / process_max_fds, not on traefik_open_connections alone. The connection metric is a subset of FD usage and will miss log files, provider sockets, and leaked backend sockets. Second, the page condition should be ratio > 95% sustained for more than 2 minutes AND still rising or not declining. A deployment that intentionally sits at 95% with stable WebSocket connections should not page; the trend condition filters that out.

Fixes

Raise the limit (the real fix for most incidents)

Use 65536 or higher for production. Where you set it depends on how Traefik runs:

# systemd unit (host-installed Traefik): systemd ignores /etc/security/limits.conf
# In the [Service] section of the unit file:
LimitNOFILE=65536

# Then reload and restart
systemctl daemon-reload
systemctl restart traefik

# Docker
docker run --ulimit nofile=65536:65536 ...

# docker-compose.yml
services:
  traefik:
    ulimits:
      nofile:
        soft: 65536
        hard: 65536

For Kubernetes, pod-level ulimit is not directly exposed in the pod spec on all runtimes; the practical options are raising the node/container runtime default or adjusting it via an entrypoint wrapper.

Verify after every change:

# Confirm the new limit took effect
grep 'Max open files' /proc/$(pgrep -f traefik)/limits

Tradeoff: raising the limit buys capacity but does not fix a leak. If FDs were leaking before, they will leak to the new ceiling, just more slowly. Pair the limit increase with ratio-based alerting so you see the climb.

Restart to clear leaked FDs (buys time, not a fix)

Restarting Traefik closes every FD it holds. If the exhaustion was caused by a leak, this immediately restores service.

# Disruptive: drops all in-flight connections
systemctl restart traefik
# or
docker restart traefik
# or, in Kubernetes
kubectl rollout restart deployment/traefik -n <namespace>

Be explicit about what this costs: all existing connections drop. With graceful shutdown configured, in-flight requests drain first, but WebSocket and other long-lived connections still terminate. Do this during an active incident to recover, then schedule the leak investigation as follow-up work. If you restart without investigating, you are on a countdown to the next identical incident.

Fix the leak (if diagnosis showed one)

  • Hung backends holding connections: configure sensible timeouts in serversTransport (dial, response header, idle connection timeouts) so Traefik gives up on backends that accept but never respond.
  • CLOSE_WAIT accumulation: backends are closing connections Traefik does not reap; check for intermediaries (firewalls, NAT gateways) silently killing idle connections, and align keep-alive timeouts so Traefik’s idle timeout is shorter than any intermediary’s.
  • Long-lived connection workloads: WebSocket and gRPC connections each pin an FD indefinitely. Baseline them separately and size the FD limit for the peak concurrent connection count, not the request rate.

Prevention

  • Set the FD limit explicitly, everywhere. Never let a production Traefik inherit a default. 1024 in a container is a scheduled outage. 65536 is the floor, not the ceiling.
  • Alert on the ratio with trend. Page at process_open_fds / process_max_fds > 95% sustained 2 minutes and rising; ticket at 80%.
  • Keep steady state below 70%. The 30% headroom absorbs connection storms: HTTP/1.1 bursts, WebSocket reconnect waves after a network blip, load balancer failovers.
  • Baseline connection count against request rate. A rising traefik_open_connections with flat request rate is your earliest leak signal, weeks before the ratio pages.
  • Verify limits after every deployment change. Container image swaps, runtime upgrades, and orchestrator changes can silently reset ulimits. The /proc/<pid>/limits check takes two seconds; automate it.
  • Size for two FDs per connection plus overhead. Client side plus backend side plus provider sockets, logs, and ACME storage. If you expect 20k concurrent connections, 65536 is barely adequate.

How Netdata helps

  • Netdata charts process_open_fds against process_max_fds per process, so the saturation ratio and its trend are visible without writing PromQL during an incident.
  • Per-second granularity catches the fast-spike pattern, where FD counts jump in under a minute, that per-minute scraping can smooth away.
  • Correlating FD usage with traefik_open_connections and entrypoint request rate on one dashboard separates “genuine load” from “leak”: FDs and connections rising together versus FDs rising alone.
  • Goroutine count (go_goroutines) alongside FD count surfaces hung-backend leaks before the ratio reaches paging territory.
  • Anomaly detection on the FD ratio flags the slow, weeks-long climb that static thresholds miss until the day it pages.
  • Post-restart, the same charts confirm whether the restart actually dropped FD usage to baseline or the leak is already rebuilding.