Logstash is logging java.io.IOException: Too many open files (EMFILE), Beats shippers report refused or reset connections, and new files are not being tailed. The JVM process is still alive, systemd still says active (running), and the monitoring API on port 9600 may still answer. Nothing looks crashed, yet the pipeline has stopped accepting new work.

That is the defining shape of file-descriptor exhaustion: a partial failure that is routinely misread as a network or disk problem. The process cannot open new sockets, new persistent queue page files, or new files for the file input, because every open of those consumes one file descriptor and the per-process limit has been reached. Existing connections and already-open files keep working, which is why some traffic continues to flow while new connections are refused.

This is also a cliff-edge resource. FD usage degrades nothing until the exact moment the limit is hit, then failures appear simultaneously across every subsystem that opens files or sockets. The good news: Logstash exposes both sides of the ratio, process.open_file_descriptors and process.max_file_descriptors, so you can see the cliff coming if you look.

What this means

Every Linux process has a per-process limit on open file descriptors (the RLIMIT_NOFILE soft limit). Logstash consumes FDs for:

  • One FD per network connection, inbound (Beats, TCP, HTTP inputs) and outbound (Elasticsearch, Kafka, HTTP outputs)
  • One FD per file actively tailed by a file input
  • Persistent queue page files and checkpoint files
  • Its own log files and various internal pipes

When open_file_descriptors reaches max_file_descriptors, the kernel refuses every new open() or accept() with EMFILE. Symptoms appear in whatever happens to need a new FD first: new Beats connections refused, PQ page allocation failing, a rotated log file that cannot be tailed. The playbook calls this exactly: “Approaching the FD limit causes new connections to fail, file opens to error, and partial system failures where the process is alive but cannot accept new work.”

flowchart TD
  A[FD consumers accumulate] --> B[open_file_descriptors climbs]
  B --> C{ratio vs max_file_descriptors}
  C -->|under limit| D[normal operation]
  C -->|hits limit| E[EMFILE: Too many open files]
  E --> F[new Beats/TCP connections refused]
  E --> G[PQ page files cannot open]
  E --> H[file input cannot tail new files]
  F --> I[partial outage, process still alive]
  G --> I
  H --> I

Common causes

CauseWhat it looks likeFirst thing to check
Default ulimit too low for the workloadmax_file_descriptors reports 1024 or another low value; exhaustion hits at modest connection countscurl http://127.0.0.1:9600/_node/stats/process and read max_file_descriptors
File input tailing too many filesThousands of files match the path glob; FD count scales with matched files, one FD per tailed fileCount files matched by the input’s path glob; check max_open_files and close_older settings
Connection churn on outputsOutput destination flapping; each reconnect cycle opens new sockets faster than old ones closeOutput error/retry patterns in /var/log/logstash/logstash-plain.log
Connection leakFD count grows monotonically over days or weeks without returning to baseline; playbook: “Connections establish but never close, FD count climbs slowly”Trend open_file_descriptors over days; compare against peak_open_file_descriptors
Beats input connection accumulationMany inbound shipper connections held open; FD count tracks shipper countss -tnp against the Logstash PID, count connections to the Beats port
PQ page files accumulatingFD usage grows with queue depth during a downstream outagePQ occupancy in /_node/stats/pipelines; page files under the queue directory

A useful early distinction: is max_file_descriptors simply too small for a legitimate workload, or is FD count growing without a corresponding workload increase? The first is a sizing fix. The second is a leak, and raising the limit only delays the next incident.

Quick checks

All read-only. Run them before changing anything.

# 1. Current FD usage and limit, straight from Logstash
curl -sS http://127.0.0.1:9600/_node/stats/process?pretty
# Read: process.open_file_descriptors, process.peak_open_file_descriptors,
# process.max_file_descriptors

# 2. Confirm at OS level and see the actual soft limit
PID=$(pgrep -f org.logstash.Logstash)
ls /proc/$PID/fd | wc -l
grep "Max open files" /proc/$PID/limits

# 3. See what the FDs actually are (sockets vs files vs pipes)
PID=$(pgrep -f org.logstash.Logstash)
ls -l /proc/$PID/fd 2>/dev/null | awk '{print $NF}' | grep -c socket
ls -l /proc/$PID/fd 2>/dev/null | grep -v socket | awk '{print $NF}' | sort | uniq -c | sort -rn | head -20

# 4. Count inbound connections per peer (Beats input pressure)
PID=$(pgrep -f org.logstash.Logstash)
ss -tnp | grep "pid=$PID" | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head

# 5. Count files matched by the file input glob
# Adjust the pattern to your configured path
ls /var/log/myapp/*.log 2>/dev/null | wc -l

# 6. Check for connection churn evidence in logs
grep -Ei '(retry|error|exception|failed|refused|timeout|reset)' /var/log/logstash/logstash-plain.log | tail -n 100

# 7. Confirm what limit systemd is enforcing
systemctl show logstash -p LimitNOFILE

Check 7 matters more than it looks: if Logstash runs under systemd, the unit’s LimitNOFILE is the limit that actually applies, regardless of what you set in /etc/security/limits.conf or via ulimit in a shell.

How to diagnose it

  1. Establish the ratio. From /_node/stats/process, compute open_file_descriptors / max_file_descriptors. Above 0.8 you are in the danger zone; at 1.0 you are in the incident. Also note peak_open_file_descriptors: if the peak is near the max but current usage has dropped, the process already hit the wall once and partially recovered.

  2. Decide: leak or sizing. Look at the FD trend over days, not minutes. A sawtooth that returns to baseline is workload-driven churn. A monotonic climb over weeks is a leak. The playbook flags the leak case as a silent catastrophe: “FD leak over weeks… Eventually crashes the process with too many open files.”

  3. Attribute the FDs. Use check 3 above. If most FDs are sockets, the problem is connection-side: too many inbound shippers, output reconnection churn, or connections never closing. If most FDs are regular files under log directories, the file input is the consumer. If they cluster under the PQ directory, queue depth during a downstream outage is the driver.

  4. For file inputs, check the internal cap. The file input plugin has its own max_open_files setting (default 4095) independent of the OS limit. If you see a “Reached open files limit” style warning in the logs while OS-level FDs are still comfortable, you are hitting the plugin’s cap, not EMFILE. The same plugin’s close_older setting (default 1 hour) controls how quickly idle tailed files release their FDs.

  5. For connection churn, correlate with output errors. FD pressure plus a rising retry/error pattern in the logs is the playbook’s “output reconnection storms creating FD churn.” Check the downstream destination’s health; a flapping Elasticsearch or Kafka forces the output to tear down and rebuild connections continuously.

  6. Verify which limit layer is binding. Compare three values: the systemd unit’s LimitNOFILE (check 7), anything set in /etc/security/limits.conf, and the max_file_descriptors the process actually reports. The process’s reported value is the truth. If you edited limits.conf and the process still reports 1024, that edit did nothing for a systemd-managed service.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
process.open_file_descriptors / process.max_file_descriptorsThe direct exhaustion ratio> 0.8 sustained (ticket)
process.peak_open_file_descriptorsShows whether you already grazed the limit during a burstPeak within ~10% of max
FD growth rate over daysSeparates leak from sizing; enables runway mathPositive trend with flat workload
Output error/retry pattern in logsReconnection churn is a top FD consumerSustained nonzero retries correlated with FD growth
PQ queue_size_in_bytes and page file countQueue depth during outages adds FDsPQ growing while FD count grows
Inbound connection count per inputBeats/TCP shipper accumulationConnection count rising without shipper fleet change

Runway estimation from the playbook’s capacity section: (max_file_descriptors - open_file_descriptors) / current_growth_rate. If growth rate is zero, there is no leak risk; if it is positive, that formula tells you how long you have.

Fixes

Raise the limit (sizing fix)

If the workload legitimately needs more FDs, raise the limit at the layer that actually applies:

# Create a systemd override for the Logstash unit
systemctl edit logstash

[Service]
LimitNOFILE=65535

systemctl daemon-reload
systemctl restart logstash

The restart is disruptive: with a memory queue, in-flight events are lost; with a persistent queue, events survive but ingestion pauses. Schedule accordingly. After restart, confirm with curl http://127.0.0.1:9600/_node/stats/process that max_file_descriptors reflects the new value.

Do not rely on /etc/security/limits.conf or shell ulimit -n for a systemd service; systemd does not apply PAM limits to units. If your deployment uses init scripts generated from startup.options, LS_OPEN_FILES there is the equivalent knob, but under systemd the unit file wins. The packaged systemd unit already ships with a raised LimitNOFILE in current packages; the exact default value varies by version.

Reduce file input FD consumption

  • Tighten the path glob. Match only the files you actually need. A wildcard that sweeps up archived or rotated files pins an FD per file.
  • Lower close_older. Files idle longer than this are closed, cycling FDs back to the pool. The default is 1 hour; if you tail many mostly-idle files, a shorter interval frees FDs faster.
  • Raise max_open_files only deliberately. The default 4095 is a guardrail. Raising it past your OS limit just moves the failure from the plugin to the kernel.

Fix connection churn

  • Stabilize the downstream. If output retries correlate with FD growth, the fix is at the destination (Elasticsearch health, Kafka brokers, network path), not in Logstash.
  • Review idle connection handling on inputs. For the Beats input, client_inactivity_timeout controls how long idle shipper connections are held; very short values cause reconnect churn, very long values pin FDs on dead peers.

Fix a leak

A monotonic FD climb with flat workload is a leak: connections or files opened and never closed. Raising the limit is a stopgap that buys runway at best. Capture ls -l /proc/<pid>/fd snapshots hours apart and diff them to find which FD category never returns to baseline, then chase that plugin. A process restart clears the FDs but not the cause, so treat restart as mitigation, not resolution, and keep the trend monitoring in place to confirm whether the climb resumes.

Prevention

  • Alert on the ratio. open_file_descriptors / max_file_descriptors > 0.8 sustained is the ticket-level threshold from the playbook. FD exhaustion gives you a long, measurable runway if you watch it; it gives you nothing if you do not.
  • Size the limit for the workload. Count expected FDs: shipper connections, tailed files, PQ pages, output connections, then double it for reconnection spikes. The playbook’s headroom definition: “Peak FD count stays well below hard limit with room for reconnection spikes.”
  • Track the trend, not just the level. A weekly review of the FD count trend catches slow leaks months before they become incidents. Flat workload plus rising FDs equals leak; act while runway is long.
  • Correlate with output health. FD pressure plus output retries is churn; FD pressure plus connection errors is exhaustion. Keep both signals on the same dashboard so the correlation is visible during incidents.
  • Audit file input globs at deploy time. A config change that broadens a path glob from hundreds to tens of thousands of files is an FD incident scheduled for later.

How Netdata helps

  • Netdata collects per-process file descriptor counts from /proc alongside the Logstash process, so FD pressure shows up as a continuous trend rather than a number you check during an incident.
  • Correlating FD count with per-process socket counts and network connection states distinguishes connection churn (sockets climbing) from file input pressure (regular files climbing) without manual ls -l /proc/<pid>/fd archaeology.
  • Plotting FD usage next to Logstash output throughput and queue metrics makes the classic pattern legible: downstream flapping, retry churn, FD growth, then refused connections.
  • A ratio alert at 80% of max_file_descriptors converts the cliff edge into a ticket with weeks of runway, which is exactly the playbook’s recommended posture for this resource.
  • Netdata’s long retention on per-process metrics is what makes slow FD leaks visible; a two-week climb is obvious on a month-long graph and invisible in a 15-minute dashboard window.