Logstash is up, throughput looks normal, and then new connections start failing, file opens error out, and the process dies with “too many open files.” The failure looks sudden. It usually is not. File descriptor pressure builds over days or weeks as process.open_file_descriptors climbs toward process.max_file_descriptors, and exhaustion is a cliff edge: everything works until the limit is hit, then new connections, new file opens, and sometimes logging itself fail at once.

The three drivers that show up again and again are slow FD leaks (connections or files opened and never closed), file inputs tailing more files than expected, and output reconnection storms that churn sockets faster than the OS reclaims them. All three are visible in the same two numbers if you look before the cliff.

What this means

Every network connection (input listeners, output connections), every tailed file, every persistent queue page file, and every log file Logstash holds open consumes one file descriptor. The process has a ceiling: process.max_file_descriptors, the soft limit the service manager set at startup. As the open count approaches it, Logstash fails in partial, confusing ways: it is alive, the API may still respond, but it cannot accept new work.

The diagnostic split that matters:

  • A stable high baseline means you are legitimately using many FDs (tailing thousands of files, many Beats connections). This is a sizing problem.
  • A slow climb that never returns to baseline means a leak. Leaks can take weeks to manifest, which is why they are missed.
  • Sharp spikes correlated with downstream trouble mean reconnection churn: an output loses its connection, reconnects aggressively, and old sockets linger in states like CLOSE_WAIT while new ones pile up.
flowchart TD
  A[File input tails many files] --> D[open_file_descriptors rising]
  B[Connection leak: opened, never closed] --> D
  C[Output reconnection storm: CLOSE_WAIT buildup] --> D
  D --> E{open / max ratio}
  E -->|under 80%| F[Watch trend, compute runway]
  E -->|over 80% sustained| G[Act now: identify FD type]
  G --> H[Cliff: too many open files, partial failure]

If FDs are growing, estimate time to exhaustion:

runway = (max_file_descriptors - open_file_descriptors) / growth_rate

A leak growing 50 FDs per day with 8,000 FDs of headroom gives you roughly 160 days. That sounds comfortable until nobody notices for 160 days.

Common causes

CauseWhat it looks likeFirst thing to check
Connection/socket leak in an input or output pluginFD count climbs steadily over days or weeks, never returns to baseline after traffic dropsls -l /proc/<pid>/fd for sockets that never disappear; correlate FD growth with connection counts
File input with broad wildcardsHigh, stable FD count; matches the number of files discovered under paths like /var/log/**/*.logCount files matched by the input path; compare against open FD count of type regular file
Output reconnection stormFD spikes during downstream outages or flapping; many sockets in CLOSE_WAITss -tan state close-wait and output error/retry lines in logstash-plain.log
Persistent queue and DLQ page filesFD count grows during queue buildup, may lag behind queue drainls -l /proc/<pid>/fd for FDs pointing at the queue and dead_letter_queue directories
Limit set too low for the workloadFD ratio sits above 80% during normal peaks with no growth trendcat /proc/<pid>/limits and the systemd unit’s LimitNOFILE

Quick checks

All read-only and safe to run during an incident.

# FD pressure straight from the Logstash API
curl -sS http://127.0.0.1:9600/_node/stats/process?pretty
# Read process.open_file_descriptors, process.peak_open_file_descriptors,
# and process.max_file_descriptors. Compute the ratio.

# OS-level view of the same thing
PID=$(pgrep -f org.logstash.Logstash)
ls /proc/$PID/fd | wc -l
grep "Max open files" /proc/$PID/limits

The limits output shows both soft and hard limits; Logstash is bounded by the soft limit. Service managers override shell defaults, so the number here is the only one that matters. On systemd systems the effective limit comes from LimitNOFILE in the unit (or an override in /etc/systemd/system/logstash.service.d/), not from an interactive ulimit.

# Classify what the FDs actually are
ls -l /proc/$PID/fd | awk '{print $NF}' | sed 's/[0-9]*$//' | sort | uniq -c | sort -rn | head -20

This buckets FDs by target: sockets, regular files under specific paths, pipes, and anon inodes. The bucket that is growing is your driver.

# Socket state breakdown: look for CLOSE_WAIT accumulation
ss -tanp | grep "pid=$PID" | awk '{print $1}' | sort | uniq -c

A large and growing CLOSE_WAIT count means the remote side closed the connection but Logstash never closed its end. That is a leak signature, and it typically points at an output plugin or a proxy in front of the destination.

# Count files matched by a broad file input path (example pattern)
find /var/log -name '*.log' -type f 2>/dev/null | wc -l

# Output retry/reconnect activity in the Logstash log
grep -Ei '(retry|reconnect|error|exception|timeout|refused)' /var/log/logstash/logstash-plain.log | tail -n 100

# Baseline trend: sample twice, 10 minutes apart
for i in 1 2; do
  curl -sS http://127.0.0.1:9600/_node/stats/process | grep open_file_descriptors
  sleep 600
done

Two samples say nothing about a multi-week leak, but during an active incident a rising delta over 10 minutes confirms active growth rather than a static high baseline.

How to diagnose it

  1. Confirm the ratio. Pull open_file_descriptors and max_file_descriptors from /_node/stats/process. Above 80% sustained, treat this as the incident, not a side note.

  2. Check the peak. peak_open_file_descriptors tells you whether you already brushed the ceiling. A peak near max with a current value well below it points at churn (spikes) rather than a leak (ratchet).

  3. Establish the trend shape. Look at FD count over the longest window you have. Three shapes: flat-and-high (sizing), slow monotonic climb (leak), sawtooth spikes tied to downstream events (reconnection churn).

  4. Classify the FDs. Use the /proc/<pid>/fd breakdown above. Sockets dominating means network paths: inputs (Beats, TCP, HTTP) or outputs. Regular files under log directories means the file input. Files under the queue directory means PQ pages.

  5. If sockets: check state and peer. Group ss -tanp output by state and by remote address. CLOSE_WAIT against your Elasticsearch or load balancer addresses, growing over time, is the classic reconnection-leak combination. Correlate the growth periods with output errors in the Logstash log.

  6. If regular files: count what the input sees. Compare the number of files your file input path patterns match against the number of open file FDs. The file input keeps one FD per actively tailed file, bounded by its max_open_files setting (default 4095) with a sliding window: it can track more files than that, but only keeps that many open at once. A “Reached open files limit” warning from filewatch in the log means the input itself is saturated and files are waiting to be opened. Idle files are closed after close_older (default 1 hour), which frees FDs for hotter files. If most matched files are constantly written, nothing goes idle and the window stays full.

  7. If churn: find the trigger. Reconnection storms follow downstream events: Elasticsearch restarts, load balancer idle timeouts cutting connections, TLS or auth failures. Line up the FD spike times with output error timestamps and with events on the destination side.

  8. Compute runway. (max - open) / growth_rate using a growth rate measured over days, not minutes. This tells you whether you have hours or weeks, and therefore whether the fix is urgent operational work or a planned change.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
process.open_file_descriptorsThe primary count. Trend shape distinguishes leak from churn from sizingAny climb that does not return to baseline after load drops
process.max_file_descriptorsThe ceiling the ratio is computed againstLower than you assumed (service manager overrides defaults)
process.peak_open_file_descriptorsReveals near-misses that current values hidePeak within 10% of max
Ratio open/maxThe alertable numberAbove 80% sustained
FD growth rateInput to the runway calculationAny sustained positive rate measured over days
Socket states (CLOSE_WAIT count)Early leak evidence before the FD total is alarmingGrowing count against one downstream peer
Output error/retry log linesReconnection churn triggerError bursts that coincide with FD spikes
File input saturationInput-side FD exhaustionfilewatch “Reached open files limit” warnings

Two traps worth naming. First, pipeline stats reset on config reload, but process FD stats do not; FD graphs survive reloads, which makes them reliable for long-window trend analysis. Second, do not poll the API faster than about every 10 seconds on a loaded instance; the stats endpoint shares the JVM with the pipeline.

Fixes

Fix the leak

A true leak (FDs opened and never closed) is almost always inside a plugin or its interaction with a proxy, not something you can tune away. Historical examples include Beats input connections leaked under backpressure, HTTP input connections leaked when clients outpaced the pipeline, and Elasticsearch output connections accumulating in CLOSE_WAIT when a proxy in the middle closed them first. The pattern to look for is FD growth that tracks connection events, not data volume.

  • Upgrade the affected plugin or Logstash version. Several FD leaks in the Beats input, HTTP input, and Elasticsearch output were fixed in specific releases. If your FD growth matches a known pattern (for example, growth only under backpressure), check the plugin changelog before anything else.
  • Remove the middlebox trigger. If CLOSE_WAIT sockets pile up against a load balancer or proxy that idles connections out, align its idle timeout with the output plugin’s connection reuse behavior, or have the output side close connections first. The side that closes first avoids the lingering half-closed state.
  • Restart as mitigation, not as a fix. A restart releases every FD and resets the clock. With a persistent queue, in-flight events survive. With a memory queue, queued events are lost. If the leak took weeks to build, a restart buys you weeks, but schedule the real fix.

Tame the file input

  • Narrow the path patterns. Replace broad wildcards like /var/log/**/*.log with explicit directories per pipeline. Split unrelated log trees into separate pipelines so one hot tree does not starve another.
  • Size max_open_files deliberately. The default 4095 is independent of the OS limit. Raising the systemd limit without raising max_open_files does nothing for a saturated file input; raising max_open_files without OS headroom just moves the ceiling. Set both, and keep the input’s value comfortably below the process limit.
  • Tune close_older for your write pattern. If files go idle quickly, a shorter close_older frees FDs sooner. If everything is hot, closing and reopening just adds churn; the real fix is fewer matched files or more headroom.
  • Consider read mode for batch workloads. In read mode the input closes files at EOF instead of holding them open, which suits content-complete files. In the default tail mode, files stay open as long as they are active. See the file input and sincedb guide linked below for the re-read and duplication tradeoffs.

Stop reconnection churn

  • Fix the downstream first. Reconnection storms are a symptom. If Elasticsearch is flapping, rejecting, or being restarted, the FD churn stops when the destination stabilizes.
  • Watch retry behavior during incidents. Output retries preserve data but multiply connection attempts. During a prolonged downstream outage, FD spikes are expected; the danger is a spike that never decays after recovery, which converts churn into a leak.
  • Check authentication and TLS. Auth failures and handshake errors cause rapid connect-fail-retry loops that churn FDs without ever establishing a working connection. The log patterns in the quick checks section catch this.

Raise the limit (with eyes open)

Raising LimitNOFILE in a systemd override (/etc/systemd/system/logstash.service.d/override.conf, then systemctl daemon-reload and a restart) is legitimate when the workload genuinely needs more FDs, for example tailing thousands of files by design. Note that the restart is disruptive: with a memory queue, in-flight events are lost. A limit increase is not a fix for a leak; it only lengthens the runway. Always pair it with growth-rate monitoring so you know whether the new ceiling is being consumed.

Prevention

  • Alert on the ratio, not the absolute count. Threshold: open_file_descriptors / max_file_descriptors above 80% sustained. Absolute counts mean different things on different deployments; the ratio travels.
  • Alert on the trend, not just the level. A leak at 30% utilization is invisible to a level alert for weeks. A growth-rate alert (sustained positive slope over days, normalized for traffic) catches leaks while runway is still long.
  • Compute and track runway. (max - open) / growth_rate turns FD monitoring from a binary alarm into a capacity signal. A runway under 30 days deserves a ticket; under 7 days, treat it as urgent.
  • Baseline FDs per config change. File input path changes, new Beats sources, and new outputs all move the legitimate baseline. Record the expected FD footprint when configs change so drift is visible.
  • Watch socket states during downstream incidents. CLOSE_WAIT growth during an outage is normal in small amounts; a count that keeps climbing after the downstream recovers is your leak alarm.

How Netdata helps

  • Netdata charts open_file_descriptors against max_file_descriptors continuously, so the ratio and the long slow leak trend are visible on one graph instead of buried in periodic manual polls.
  • Long retention at per-second granularity is what makes week-scale leaks diagnosable: you can zoom from a month-long climb down to the hour the slope changed and line it up with a deploy or a downstream incident.
  • peak_open_file_descriptors alongside the current value surfaces near-misses from reconnection spikes that a current-value-only check would miss.
  • Correlating the FD curve with process CPU, JVM heap, and pipeline throughput on the same host timeline separates the three drivers: leak (FDs up, everything else flat), churn (FD spikes aligned with output errors), and sizing (FDs tracking input rate).
  • Alerting on the ratio with a sustained-duration condition gives you the >80% ticket signal the playbook recommends, without paging on brief reconnection spikes.