Filebeat has stopped delivering logs. Its harvesters are running, files are being read, but the registry on the Filebeat host has not advanced in twenty minutes and downstream dashboards are going stale. Or the opposite variant: Filebeat’s log is a rotating wall of “Failed to publish events” and “connection reset by peer” while Logstash looks perfectly healthy on process checks.

The Beats input is the front door to Logstash. Filebeat speaks the Lumberjack protocol over TCP, conventionally on port 5044, and that connection is where Logstash-side trouble shows up first. When Logstash cannot drain its queue fast enough, the backpressure does not stay inside Logstash. It propagates out the Beats input, across the TCP connection, and into Filebeat, where it appears as a stalled registry, growing disk spool, or publish errors.

The trap is that this is a two-sided failure. Logstash can look “up” (process alive, API answering on 9600) while every connected Filebeat is wedged. This guide walks through how the backpressure propagates, how to tell a queue-driven stall from a connection-level break, and which signals on both ends pin down the cause.

What this means

The Lumberjack protocol is window-based. Filebeat sends a batch of events, then waits for ACKs from Logstash before sending more. Logstash controls the ACK pace, so it controls Filebeat’s send rate. Two distinct mechanisms throttle or sever that flow, and they have different root causes:

  1. Backpressure (flow control, connection stays up). When the pipeline queue is full, or workers are not draining it, input threads block trying to push events into the queue. The Beats input stops reading from the socket. The TCP receive window fills, ACKs slow or stop, and Filebeat’s window stalls. Filebeat is connected but cannot send, so its registry stops advancing. This is Logstash saying “slow down”, and it is almost always a symptom of a downstream problem (slow Elasticsearch, CPU-bound filters, GC pressure, or a full persistent queue), not a problem with the Beats input itself.

  2. Connection failure (flow stops entirely). The listener is gone (port conflict, pipeline failed to start), the TLS handshake fails (config mismatch, expired cert, renamed SSL settings after an upgrade), or something in the middle kills the session (firewall or load balancer idle timeout causing “connection reset by peer”). Filebeat reconnects, fails again, and cycles.

flowchart LR
  FB[Filebeat agents] -->|Lumberjack, TCP 5044| BI[Beats input]
  BI --> Q[Pipeline queue]
  Q --> W[Worker threads]
  W --> F[Filters]
  F --> O[Output: Elasticsearch etc.]
  O -.->|slow or rejecting: ACK delay| W
  W -.->|queue not draining| Q
  Q -.->|queue full: input threads block| BI
  BI -.->|socket not read: TCP window fills| FB

Distinguishing these two is the first diagnostic fork. A stalled-but-established connection points at pipeline backpressure. Refused, reset, or handshaking connections point at listener, TLS, or middlebox problems.

Common causes

CauseWhat it looks likeFirst thing to check
Downstream backpressure cascadeConnections to 5044 established, Filebeat registry frozen, Logstash queue growing, CPU low-to-moderateflow.queue_backpressure and output plugin errors/retries
CPU-bound filters (grok hell)Same stall shape, but Logstash CPU is pegged and workers are saturatedflow.worker_utilization and per-filter duration_in_millis
GC death spiralFilebeat stalled, Logstash API slow or intermittently unresponsive, throughput wobbling to zeroJVM heap post-GC floor and old-gen GC time
Persistent queue fullPQ occupancy at max_bytes, all inputs blocked, long downstream outage behind itqueue.queue_size_in_bytes vs max_queue_size_in_bytes
Listener down or port conflictFilebeat sees “connection refused”, nothing accepting on 5044ss -tlnp for the listener and Logstash startup logs
TLS mismatch or expired certConnect then immediate failure; handshake errors on one or both sidesLogstash log for SSL/handshake errors; ssl_enabled on both ends
Firewall or LB idle timeout“connection reset by peer” after idle periods, especially through a load balancerFilebeat log error strings; idle timeout on the middlebox
Too many Beats connectionsNew connections fail, “too many open files” in logs, FD count near limitprocess.open_file_descriptors vs max_file_descriptors

Quick checks

All of these are read-only and safe to run during an incident.

# 1. Is the Beats listener actually up and accepting?
ss -tlnp | grep 5044

# 2. How many Beats connections are established, and from where?
ss -tn state established '( sport = :5044 )' | wc -l
# Column 5 is the peer (client) address:
ss -tn state established '( sport = :5044 )' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head

# 3. Is Logstash backpressuring its inputs right now?
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty
# Look at: flow.queue_backpressure, flow.input_throughput, flow.output_throughput,
# queue.events_count, events.queue_push_duration_in_millis

# 4. Is the per-input event rate for the beats input moving?
# Take two samples of plugins.inputs[].events.out for the beats input, 30s apart, and compare.

# 5. Are workers blocked on outputs or burning CPU in filters?
curl -sS 'http://127.0.0.1:9600/_node/hot_threads?pretty'

# 6. TLS or authentication failures on the input?
grep -Ei '(SSL|TLS|certificate|handshake)' /var/log/logstash/logstash-plain.log | tail -n 50

# 7. File descriptor headroom (each Beats connection is a socket):
curl -sS http://127.0.0.1:9600/_node/stats/process?pretty
# Compare process.open_file_descriptors to process.max_file_descriptors

On the Filebeat side, read the Filebeat log for the last publish errors. “Failed to publish events caused by: read tcp … connection reset by peer” points at a severed session (middlebox, restart, or overload), while long silences with no publish activity and a frozen registry point at backpressure. Also confirm whether Filebeat’s own queue is growing, which tells you it has data it cannot ship.

How to diagnose it

  1. Fork on connection state. If ss shows zero established connections to 5044 and Filebeat reports “connection refused”, the listener is down: check that the pipeline with the beats input is present and running (/_node/stats/pipelines), and read the Logstash startup log for port binding or plugin initialization errors. If connections exist but Filebeat cannot send, it is flow control, continue below.

  2. Confirm backpressure inside Logstash. Check flow.queue_backpressure (8.x; the fraction of time input threads spend blocked pushing into the queue) and events.queue_push_duration_in_millis. Values materially above the pipeline’s baseline confirm the Beats input stall is queue-driven, not connection-driven. Note that the beats input runs in its own thread; when it blocks, every Filebeat connected to it stalls together.

  3. Find what is blocking the queue. Correlate three things:

    • Queue growing + output errors or retries in the logs = downstream problem (Elasticsearch rejections, timeouts, 429s). CPU will be moderate because workers wait on I/O.
    • Queue growing + CPU pegged + hot threads in grok/ruby = compute bottleneck.
    • Queue growing + post-GC heap floor high + old-gen GC frequent = GC death spiral. See queue full: inputs blocked and the backpressure wedge for the full wedge mechanics.

  4. If the queue is persisted, check occupancy and runway. Compare queue.queue_size_in_bytes to max_queue_size_in_bytes. A PQ at 90%+ with output rate below input rate means the Beats input block becomes a hard stop soon. Estimate runway as (max_queue_size_in_bytes - queue_size_in_bytes) / fill_rate. See persistent queue full: max_bytes reached and inputs blocked.

  5. If connections are resetting, identify who resets. “Connection reset by peer” during idle periods, especially with a firewall or load balancer between Filebeat and Logstash, is the classic middlebox idle-timeout signature. The Beats input’s client_inactivity_timeout (default 60 seconds) also closes genuinely idle connections, which is normal and self-healing; repeated resets under active load are not. Raising client_inactivity_timeout does not fix firewall-driven resets, a commonly reported dead end.

  6. If the listener is up but handshakes fail, check TLS config on both ends. SSL is disabled by default (ssl_enabled: false), so a Filebeat configured for TLS against a plaintext listener (or the reverse) fails immediately. When SSL is enabled, the supported protocols default to TLSv1.2 and TLSv1.3. After upgrades, check for renamed settings: in the Beats input 7.0.0, ssl became ssl_enabled, ssl_verify_mode became ssl_client_authentication, tls_min_version/tls_max_version became ssl_supported_protocols, and old setting names cause plugin startup failure. Older Filebeat configs used tls.* keys where newer versions expect ssl.*. A mismatch here severs the input entirely.

  7. Watch for the stuck-Filebeat failure mode. There is a reported bug (elastic/beats#16335, filed against Filebeat 7.5.2) where Filebeat enters a permanent connect, send, i/o timeout, disconnect cycle after a Logstash overload, even after Logstash recovers, and only a Filebeat restart clears it. If Logstash is healthy again but one Filebeat still cycles every ~30 seconds, suspect this before suspecting the network.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
flow.queue_backpressure (per pipeline, 8.x)Direct measure of input throttling; this is what Filebeat feelsSustained rise above pipeline baseline
events.queue_push_duration_in_millisEarliest backpressure signal, before the queue visibly fillsSustained non-trivial per-event wait
queue.events_count / PQ occupancyBacklog between Filebeat and the outputsMonotonic growth; PQ >80% of max_bytes
flow.input_throughput vs flow.output_throughputWhether the pipeline is keeping up with what Filebeat sendsInput > output sustained >15 min
Per-input plugins.inputs[].events.outIsolates the beats input from other inputs in the same pipelineBeats input flat while others move
Established connections to 5044Fleet-wide connection health; a drop means mass disconnectsCount far below expected agent count
process.open_file_descriptors vs maxEach Beats connection consumes a socket; exhaustion blocks new agentsRatio >80%
Output errors/retries in logsThe usual root cause behind the backpressureSustained 429s, timeouts, rejects
JVM heap post-GC floor, old-gen GC timeGC spirals stall ACKs while the process looks aliveRising floor, old-gen collections frequent

Fixes

Queue-driven backpressure

Fix the thing blocking the queue, not the Beats input. If the output is the bottleneck, restore downstream health first; Logstash will drain and release Filebeat on its own. If filters are CPU-bound, reduce filter cost or add workers only where CPU headroom exists. If the PQ is absorbing a long downstream outage and runway is short, shed non-critical sources early rather than letting the queue hit max_bytes and hard-block every agent at once. Do not restart Logstash as a first move: with a memory queue you lose the backlog, with a PQ you add replay and checkpoint risk, and neither addresses the cause.

Filebeat-side stuck loop

If Logstash has recovered but a specific Filebeat is still in a reconnect/timeout cycle (the known bug above), restart that Filebeat. It is disruptive but bounded: Filebeat resumes from its registry. Before restarting en masse, confirm Logstash truly is healthy (flow.queue_backpressure back to baseline), or the restarted agents will just re-wedge.

Firewall or load balancer resets

Set ttl in Filebeat’s Logstash output to a value below the middlebox’s idle timeout so Filebeat cycles the connection before the firewall kills it. Note the constraint from the official docs: pipelining must be set to 0 when ttl is used, because pipelining is incompatible with it. Tradeoff: more frequent reconnects, slightly more connection churn on the input.

TLS and listener issues

Align ssl_enabled and protocol versions on both ends, rotate expired certificates, and after any upgrade grep the Logstash startup log for plugin errors caused by renamed SSL settings. If the pipeline failed to start because of an invalid beats input setting, the listener never binds, and the only symptom on the Filebeat side is “connection refused”.

Connection scaling

With multiple Logstash instances, set loadbalance: true in Filebeat’s output so connections spread across hosts; the default is false (one host at a time with failover). Be aware of a community-reported behavior where, after a significant backpressure event, Filebeat appears to bias toward one Logstash node even with loadbalancing enabled, leaving that node’s queue fuller than its peers. If you see per-node imbalance after an incident, restarting the affected agents or the hot node rebalances. Also keep FD headroom: the established connection count to 5044 plus file inputs plus PQ pages must stay well under max_file_descriptors.

Prevention

  • Monitor both ends of the pipe. Filebeat’s frozen registry and Logstash’s flow.queue_backpressure are the same event seen from two sides. Alerting on only one side doubles diagnosis time.
  • Track per-input rates, not just pipeline aggregates. In a multi-input pipeline, a dead or stalled beats input hides inside a healthy-looking average.
  • Alert on PQ runway, not just occupancy. The question during every downstream incident is “how long until the Beats input blocks?”, and the fill rate answers it.
  • Baseline connection counts to 5044 so a mass disconnect (TLS expiry, LB change) is immediately visible.
  • Pin down middlebox timeouts on any path between agents and Logstash, and set Filebeat ttl accordingly before the first incident, not after.
  • Test upgrades against renamed SSL settings in staging; an old ssl => true in a 7.0+ beats input config takes the whole listener down at startup.

How Netdata helps

  • Netdata collects the Logstash node stats API continuously, so queue_backpressure, input/output throughput, and queue depth are visible as time series rather than incident-time curl snapshots.
  • Correlating queue backpressure with per-input event rates shows the exact moment the beats input started throttling, and whether other inputs stalled with it.
  • JVM heap and GC charts alongside pipeline flow metrics separate the GC-spiral stall from the downstream-blockage stall without capturing hot threads by hand.
  • File descriptor usage trending against the process limit catches slow connection accumulation from growing Beats fleets before new agents start failing.
  • Because Filebeat-side symptoms and Logstash-side causes land on the same dashboard, the two-sided nature of this failure stops being a blind spot.