Logstash Operations Guides on Netdata

How Logstash actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

Most Logstash incidents are diagnosed badly because the operator is reasoning about the wrong system. They see a “log shipper” and check whether the process is up. The process is up. The API returns 200. Nothing is being delivered. Or they see high CPU and assume the host is undersized, when the real problem is one grok pattern with catastrophic backtracking. Or they watch queue depth stay flat for hours and conclude all is well, while the persistent queue quietly absorbs a downstream outage that will page someone at 4 a.m. when it fills.

Logstash _dateparsefailure: timestamp formats that stop parsing

Mon, 01 Jan 0001 00:00:00 +0000

The _dateparsefailure tag appears when Logstash’s date filter cannot parse a timestamp field against any of its configured match patterns. The event still flows through the pipeline and reaches its output. @timestamp stays at ingestion time instead of being set to the event’s actual time.

The result: time-based dashboards show gaps or misplaced data points, index lifecycle management operates on ingestion time so events land in the wrong time bucket or expire at the wrong time, and the issue can persist for days because throughput metrics stay green. The date filter tries each format string in the match array sequentially. The first successful parse wins. If none succeed, the filter appends its tag_on_failure tag (default _dateparsefailure) and leaves @timestamp unchanged.

Logstash _grokparsefailure: why grok stops matching and how to fix it

Mon, 01 Jan 0001 00:00:00 +0000

Events tagged with _grokparsefailure pass through your pipeline unstructured. Grok could not match any of its configured patterns against the event’s input field, so it appended the failure tag and moved on. The event is not dropped. It is not counted as filtered. Unless you conditionally route it, it reaches your output destination carrying raw, unparsed data alongside the failure tag.

Throughput metrics stay green because events are still flowing. The process is up, workers are busy, queue depth is normal, and output counts look healthy. But the data in your indices is wrong: fields that downstream dashboards, alerts, and searches depend on are missing or empty because grok never extracted them. The grok filter’s per-plugin failures counter is the most direct signal for this problem and rarely has dedicated monitoring.

Logstash _jsonparsefailure: malformed JSON and codec mismatches

Mon, 01 Jan 0001 00:00:00 +0000

When events arrive at your Elasticsearch indices carrying the _jsonparsefailure tag, they contain raw, unparsed data instead of the structured fields your downstream consumers expect. Throughput metrics look healthy, events-out counts keep climbing, and no alerts fire. But the data is wrong.

The _jsonparsefailure tag is added by either the json codec or the json filter when it receives input it cannot parse as valid JSON. The event is not dropped. It flows through to the output carrying whatever raw data was received, plus the failure tag. This is a silent correctness failure, not an availability failure.

Logstash address already in use: input port conflicts on Beats, TCP, and HTTP

Mon, 01 Jan 0001 00:00:00 +0000

Logstash logs Address already in use (a Java BindException) when a listening input plugin, typically the Beats input on 5044, a tcp, http, or syslog input, tries to bind a port that is already taken. The bind fails, the input cannot start, and the pipeline that owns it stops ingesting.

The blast radius is the problem. In a multi-pipeline deployment the JVM usually stays alive and the other pipelines keep running. Process-level checks stay green, the monitoring API answers, and one pipeline silently stops ingesting. Upstream Filebeat agents queue, syslog senders drop, and nobody notices until someone asks where a stream of logs went.

Logstash API unreachable on port 9600: crash, GC pause, or startup

Mon, 01 Jan 0001 00:00:00 +0000

Your liveness check against http://127.0.0.1:9600/ just started timing out or returning connection refused. That single fact tells you almost nothing by itself. The Logstash monitoring API shares the JVM with the pipeline, so an unreachable API means one of four things: the process is dead, the JVM is frozen in a garbage collection pause, Logstash is still starting up, or something is wrong with how the API is bound to the network.

Logstash Beats input: Filebeat backpressure and connection health

Mon, 01 Jan 0001 00:00:00 +0000

Filebeat has stopped delivering logs. Its harvesters are running, files are being read, but the registry on the Filebeat host has not advanced in twenty minutes and downstream dashboards are going stale. Or the opposite variant: Filebeat’s log is a rotating wall of “Failed to publish events” and “connection reset by peer” while Logstash looks perfectly healthy on process checks.

The Beats input is the front door to Logstash. Filebeat speaks the Lumberjack protocol over TCP, conventionally on port 5044, and that connection is where Logstash-side trouble shows up first. When Logstash cannot drain its queue fast enough, the backpressure does not stay inside Logstash. It propagates out the Beats input, across the TCP connection, and into Filebeat, where it appears as a stalled registry, growing disk spool, or publish errors.

Logstash certificate expiry: the silent, total outage no built-in metric shows

Mon, 01 Jan 0001 00:00:00 +0000

Every TLS connection into or out of Logstash depends on a certificate with a hard expiry date. When that date passes, the failure is not gradual: every new TLS handshake is rejected, every client disconnects, and every output stalls at the exact second the certificate lapses. Filebeat agents stop shipping. Elasticsearch outputs throw handshake errors. Nothing is delivered.

The dangerous part is what Logstash does not tell you. The Node Stats API exposes JVM, pipeline, event, queue, and plugin metrics, but there is no certificate expiry date, no days-until-expiry gauge, and no TLS handshake failure counter anywhere in it. Your dashboards can be completely green at T-minus one hour and show a total ingestion outage at T-plus zero. Certificate expiry is a property of files on disk, so it needs an external check, not a Logstash metric.

Logstash config reload failed: reloads.failures and invisible configuration drift

Mon, 01 Jan 0001 00:00:00 +0000

You pushed a config change. The reload counter incremented, the process never restarted, and nothing paged. Two days later someone notices the new filter logic is not in the data. The deploy “worked” in every system that tracks deploys, but Logstash is still running the old pipeline.

With config.reload.automatic enabled, Logstash polls config files, validates changes, and hot-swaps the pipeline. When validation fails, it keeps the old pipeline running. That is the safe choice, but it means the config on disk and the config in memory diverge with no process-level symptom. The only evidence lives in three counters almost nobody graphs: reloads.failures, reloads.last_error, and reloads.successes.

Logstash configuration drift: when the running config no longer matches the deployed one

Mon, 01 Jan 0001 00:00:00 +0000

You pushed a config fix to /etc/logstash/conf.d/ an hour ago. The pipeline is running, throughput is normal, no alert fired. But the fix never took effect: the reload failed validation, Logstash kept the old pipeline running, and nothing in your standard monitoring noticed. The running config and the deployed config are now two different things.

This is Logstash configuration drift, and it is dangerous because it is invisible by default. Logstash has no built-in drift detection: there is no API endpoint that returns the currently active config text or a hash of it. The reload machinery is deliberately safe (a failed reload keeps the old pipeline alive rather than dropping it), but that safety creates a silent gap between what you think is deployed and what is actually processing your events.

Logstash configuration integrity: detecting unexpected changes to pipeline files

Mon, 01 Jan 0001 00:00:00 +0000

Logstash has no built-in file integrity monitoring. It will load, reload, and run whatever it finds in /etc/logstash/conf.d/, pipelines.yml, and logstash.yml, and it will not tell you that those files changed outside your deployment pipeline. A hand edit at 02:00, a config-management agent fighting your last deploy, or an unauthorized change all look identical to the process: new config, reload attempt, keep running.

The risk is not just that something changed. It is that the change is silent and the effects are subtle. A modified output destination routes data somewhere it should not go. A tweaked conditional drops fields you depend on downstream. A new input opens a listen port nobody approved. Throughput metrics stay green, because the pipeline is working fine. It is just doing the wrong thing.

Logstash could not be started: another instance is using the configured data.dir

Mon, 01 Jan 0001 00:00:00 +0000

Logstash exits immediately at startup with an error like:

Logstash could not be started because there is already another instance
using the configured data.dir. Please change the value of path.data or
configure a different instance.

The wording varies slightly by version, but the meaning is always the same: Logstash tried to take an exclusive lock on its data directory and failed. This is a hard startup block. No pipeline loads, no events flow, the process exits.

Logstash CPU-bound filters (grok hell): high CPU, saturated workers, growing queue

Mon, 01 Jan 0001 00:00:00 +0000

Worker utilization is pinned above 90%. Host CPU is near its ceiling on allocated cores. The queue is growing, per-event processing duration is climbing, and output errors are absent. The pipeline is not blocked downstream. It is starved for compute because one or more filter plugins cannot keep up with the event rate.

The usual culprit is complex grok regex evaluation, though Ruby filters, heavy JSON manipulation, and enrichment plugins can produce the same signature. The critical diagnostic fork: if CPU is high, you have a compute problem. If CPU is low or moderate with the same queue growth, you have backpressure or I/O blocking, and the fixes are entirely different. This article covers the compute-bound path.

Logstash dead letter queue growing: DLQ diversion, replay, and disabled-by-default risk

Mon, 01 Jan 0001 00:00:00 +0000

A growing dead_letter_queue.queue_size_in_bytes means an output is permanently rejecting events and diverting them to the DLQ instead of delivering them downstream. This is a data-integrity failure, not just a metric ticking up. Every event in the DLQ is data your destination never received.

Two design properties compound the risk. The DLQ is disabled by default in all Logstash versions; many production deployments run without it, meaning permanently failed events are logged and silently lost with no on-disk record. And the DLQ only captures specific output failure classes. Filter and parse failures (grok, json, date) do not go to the DLQ. They pass through the pipeline with _grokparsefailure or similar tags and reach the output as malformed events.

Logstash disk full: PQ, DLQ, and log volumes competing for space

Mon, 01 Jan 0001 00:00:00 +0000

Logstash is down, the logs show disk write failures, and the persistent queue metrics look innocent: queue_size_in_bytes is well under max_queue_size_in_bytes. The queue never filled. The disk did.

This is a distinct outage path from queue fullness. queue.max_bytes limits how much the persistent queue itself allocates. It says nothing about total disk consumption on the partition. When the PQ shares a filesystem with the dead letter queue, Logstash’s own log files, or the OS, any of those consumers can fill the partition while the PQ stays comfortably within its configured limit. Once the filesystem returns no-space errors, page writes fail, checkpoint writes fail, and the process dies.

Logstash downstream backpressure cascade: when a slow output stalls the whole pipeline

Mon, 01 Jan 0001 00:00:00 +0000

Logstash is up, the API returns 200, and your dashboards are going dark. Beats agents are buffering, Kafka consumer lag is climbing, and throughput collapsed while CPU sits at 15%. The process looks healthy from the outside. It is not.

A slow or failing output (Elasticsearch, Kafka, HTTP endpoint, syslog receiver) raises output duration. Worker threads block waiting for acknowledgment. The queue fills. The persistent queue absorbs the gap for a while, sometimes hours. When it hits max_bytes, inputs are blocked. Upstream systems back up or drop events.

Logstash Elasticsearch 429: retrying failed action with response code 429

Mon, 01 Jan 0001 00:00:00 +0000

The log line retrying failed action with response code: 429 from logstash.outputs.elasticsearch means Elasticsearch refused a bulk request because its bulk thread pool queue was full (es_rejected_execution_exception). ES is saturated and cannot keep up with the incoming bulk request rate.

This is the most common trigger of the backpressure wedge in Logstash-to-ES pipelines. The ES output plugin retries the rejected batch with exponential backoff. Worker threads block on those retries. The internal queue fills. Inputs are backpressured, and upstream systems like Filebeat buffer locally or begin dropping events.

Logstash Elasticsearch mapping conflict: field type mismatches and rejected documents

Mon, 01 Jan 0001 00:00:00 +0000

A mapping conflict occurs when Logstash sends a document whose field type does not match the index mapping. A field mapped as long arrives as a string, or an object appears where a scalar was expected. Elasticsearch rejects the document at the bulk API level.

The rejection is often invisible. Elasticsearch returns HTTP 200 for bulk requests that contain per-document failures. Logstash counts the batch as events.out and moves on. If the Dead Letter Queue (DLQ) is enabled, rejected documents divert there. If it is not (the default), documents are logged at WARN and silently lost. Pipeline throughput stays green while data disappears.

Logstash Elasticsearch partial bulk failure: HTTP 200 with per-document errors and silent data loss

Mon, 01 Jan 0001 00:00:00 +0000

Your Logstash pipeline looks healthy. The process is up, the API responds on port 9600, events.out is climbing steadily, and the queue is empty. But someone asks why last Tuesday’s application logs are missing from Elasticsearch. You run a count query against the index and the numbers do not add up. Logstash says it delivered 50 million events. Elasticsearch shows 48 million documents. Two million vanished.

The Elasticsearch bulk API returns HTTP 200 as long as it successfully received and processed the request at the transport level. Individual documents within that batch may still fail with mapping conflicts, type errors, or version conflicts. Logstash counts the batch as delivered. The events.out counter goes up. The queue stays flat. Every standard monitoring signal reads green. But some documents were rejected at the index level and either went to the Dead Letter Queue (disabled by default), were logged and dropped, or were silently discarded.

Logstash event duplication: in/out ratio drift, clones, and re-read sources

Mon, 01 Jan 0001 00:00:00 +0000

Logstash exposes three cumulative event counters at the pipeline level: events.in, events.out, and events.filtered. Their relationship reflects the pipeline’s intended transformation. A passthrough pipeline should see out approximately equal to in over any sustained window. A pipeline with drop {} filters should see out plus filtered approximately equal to in. Clone and split filters legitimately produce more output events than input events.

When this ratio drifts from what the configuration intends, the pipeline is duplicating events, losing events, or both. The worst case is silent duplication: the process is up, throughput looks healthy, and the only evidence is downstream indices containing more documents than the source produced. Clone, split, aggregate, and drop filters all change the ratio by design, so the diagnostic question is not whether the ratio differs from 1:1 but whether it matches what this specific pipeline’s configuration should produce.

Logstash file descriptor pressure: leaks, tailed files, and reconnection churn

Mon, 01 Jan 0001 00:00:00 +0000

Logstash is up, throughput looks normal, and then new connections start failing, file opens error out, and the process dies with “too many open files.” The failure looks sudden. It usually is not. File descriptor pressure builds over days or weeks as process.open_file_descriptors climbs toward process.max_file_descriptors, and exhaustion is a cliff edge: everything works until the limit is hit, then new connections, new file opens, and sometimes logging itself fail at once.

Logstash file input and sincedb: re-read loops, duplicates, and FD pressure

Mon, 01 Jan 0001 00:00:00 +0000

Your Logstash process looks healthy. The API answers, the pipeline is running, workers are busy. Then someone downstream asks why the same log lines appear three times in Elasticsearch, or why events.in just spiked to ten times its baseline, or why Logstash is logging “too many open files” while traffic looks normal.

All three symptoms usually trace back to the same component: the file input plugin and its position-tracking file, the sincedb. The file input holds one file descriptor per tailed file and records how far it has read in a sincedb file on disk. When that tracking breaks (corruption, inode reuse after rotation, a remounted filesystem changing device numbers, a wildcard that suddenly matches thousands of files) the failure shows up as duplicate events, unexpected events.in spikes, or climbing process.open_file_descriptors.

Logstash flow.queue_backpressure: the input-throttling metric explained

Mon, 01 Jan 0001 00:00:00 +0000

You opened the Logstash node stats API during an incident, found flow.queue_backpressure sitting at 0.9, and now you need to know what that number is actually telling you. The short answer: it is the fraction of time your input threads spend blocked trying to push events into the pipeline queue. It measures ingestion throttling, not worker-side slowness, and that distinction drives the entire diagnosis.

This article explains what the metric counts, how it is derived, why its absolute value is less meaningful than its movement, and how to combine it with worker utilization and queue depth to decide whether you are looking at a downstream outage, a CPU-bound filter chain, or simple capacity exhaustion.

Logstash GC death spiral: high garbage-collection overhead and collapsing throughput

Mon, 01 Jan 0001 00:00:00 +0000

The Logstash process is up. systemd reports it as active. But event throughput has collapsed to near zero, and CPU is pinned at full utilization. This is a garbage-collection death spiral: the JVM spends most of its wall-clock time in GC, reclaiming almost nothing, while the pipeline starves for compute.

The failure is misleading because the process appears alive. Process-liveness checks pass. The monitoring API on port 9600 may still respond, albeit slowly. But no useful work is happening. The JVM has entered a feedback loop: each GC cycle reclaims less than the last, so GC runs more frequently, steals CPU from event processing, causes events to accumulate, fills the heap faster, and triggers even more GC.

Logstash grok slow: catastrophic backtracking and per-event duration spikes

Mon, 01 Jan 0001 00:00:00 +0000

A Logstash pipeline that ran fine for months suddenly pegs CPU. Workers saturate, the persistent queue grows, and event throughput collapses. The input rate is unchanged, the output destination is healthy, and there are no error storms in the logs. The only visible anomaly is that one grok filter’s per-event processing duration has exploded.

This is the signature of catastrophic regex backtracking, also known as ReDoS. A single malformed or near-miss log line hits a pattern with nested unbounded quantifiers or overlapping alternation, and the Oniguruma regex engine that grok uses takes exponential time to determine that the match fails. One bad event can occupy a pipeline worker thread for seconds or longer.

Logstash heap usage high: why the post-GC floor matters more than the peak

Mon, 01 Jan 0001 00:00:00 +0000

Logstash operators hit a recurring trap with JVM heap monitoring. They set an alert for heap_used_percent above 80%, following standard JVM guidance. The alert fires constantly because the JVM heap naturally cycles through allocation and collection. They silence it or raise the threshold. Then a real memory leak or GC death spiral develops, heap stays elevated for hours, and nobody notices because the alert was muted.

The problem is not the threshold. It is the metric choice. heap_used_percent is an instantaneous snapshot that catches a random point in the GC cycle. A JVM doing real work allocates aggressively between collections, so the peak naturally approaches the heap ceiling before GC reclaims dead objects. Alerting on the peak is alerting on normal behavior.

Logstash input rate dropped to zero: upstream failure vs blocked inputs

Mon, 01 Jan 0001 00:00:00 +0000

The Logstash input rate metric (flow.input_throughput or the events.in counter delta) has dropped to zero. Before restarting anything, answer one question: is the queue empty or full?

An input rate of zero with an empty queue means events are not arriving from upstream. An input rate of zero with a queue at capacity means Logstash has applied backpressure to its own inputs because it cannot drain events fast enough. These conditions require opposite responses, and treating one as the other wastes time during an incident.

Logstash JVM heap sizing: -Xms/-Xmx, the 1GB default, and why they should match

Mon, 01 Jan 0001 00:00:00 +0000

Logstash ships with a 1GB JVM heap (-Xms1g -Xmx1g in config/jvm.options), unchanged through 7.x, 8.x, and 9.x. That default works for trivial pipelines. It is catastrophically small for production and is the single most common cause of GC death spirals, throughput collapse, and OOM kills.

The fix: increase the heap, set minimum and maximum to the same value, and account for off-heap memory. Set the heap too large and you starve the OS and risk longer full-GC pauses. Set it without understanding off-heap allocation and you get OOM kills with a heap that looks comfortably under capacity. Ignore container cgroup limits and the JVM sizes itself against host RAM, not the container limit.

Logstash Kafka input: consumer group lag and rebalances

Mon, 01 Jan 0001 00:00:00 +0000

The consumer group lag for your logstash group is climbing and not draining, or the Kafka input periodically drops to zero for seconds or minutes at a time while the brokers look fine. In both cases the evidence lives outside Logstash: consumer-group lag is a Kafka-side signal, and you will not find it in the Logstash monitoring API on port 9600.

Rising lag means one thing: Logstash is consuming slower than producers are writing. Frequent rebalances mean the group cannot hold a stable assignment, and every rebalance is a window where consumption stops entirely. The two often appear together, because the most common rebalance trigger (slow event processing) is also the most common lag trigger.

Logstash memory queue vs persistent queue: durability, visibility, and failure modes

Mon, 01 Jan 0001 00:00:00 +0000

Every Logstash pipeline has a queue between its inputs and its worker threads. The default is the memory queue: a small, bounded, in-memory buffer with no durability. The alternative is the persistent queue (PQ): a page-based, checkpointed, on-disk buffer that survives restarts.

This choice changes three things that matter operationally: what you lose when the process dies, what you can see in the metrics API, and how the pipeline fails when the queue fills. Teams enable PQ for durability and then discover it added disk I/O saturation, page corruption, and a false sense of safety to their failure catalogue. Teams running the memory queue often have no written answer for what a crash costs them.

Logstash monitoring API exposed: port 9600 on a routable interface without auth

Mon, 01 Jan 0001 00:00:00 +0000

Every Logstash node runs an HTTP monitoring API, by default on port 9600. It answers questions like “what pipelines are running”, “what plugins and versions are installed”, and “show me the hottest threads with stack traces”. By default it binds to 127.0.0.1, which is safe. Exposure happens when someone sets api.http.host to 0.0.0.0 or a routable IP to make remote monitoring easier, and leaves it there.

On Logstash versions before 8.x, the API has no built-in authentication at all: anyone who can reach the port can read everything it serves. On 8.x, basic auth and SSL exist but are off by default, so an exposed bind is still unauthenticated unless you explicitly configured it.

Logstash monitoring checklist: the signals every production pipeline needs

Mon, 01 Jan 0001 00:00:00 +0000

Most Logstash monitoring setups answer the wrong question. They answer “is the process running?” when the question that matters is “are events leaving the pipeline?” A Logstash JVM can be alive, healthy by systemd’s standards, and returning 200 from its monitoring API while the queue is full, workers are blocked on a dead Elasticsearch, and zero events have been delivered for an hour. Process liveness is necessary. It is nowhere near sufficient.

Logstash monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Most Logstash deployments are monitored at the wrong level. Teams alert on process liveness and heap percentage, then get paged by users asking where Wednesday’s logs went. The process was up, the API returned 200, and throughput looked fine the whole time. The gap is not tooling. It is which signals the team decided to watch.

This article defines a four-level maturity model for Logstash monitoring: Survival, Operational, Mature, and Expert. Each level names the specific signals to collect, why they matter, and what class of failure becomes visible that was invisible at the level below. Use it as an audit checklist against your current setup, and as a roadmap for what to add next. The levels are cumulative: every level assumes everything below it is already in place.

Logstash multi-pipeline monitoring: why aggregate stats hide a failed pipeline

Mon, 01 Jan 0001 00:00:00 +0000

You migrated to pipelines.yml to isolate workloads: one pipeline per team, per source, or per destination. Each pipeline got its own queue, its own workers, its own failure domain. Right call for fault isolation. But if your monitoring still polls the node once and looks at global event counts, you have given back the visibility the isolation bought you.

The concrete scenario: five pipelines, roughly equal traffic, one of them wedges. An output stalls, a config reload fails, a Kafka consumer group gets stuck rebalancing. That pipeline’s throughput goes to zero; the other four keep flowing. Aggregate node throughput drops by about 20 percent. If your alert is “throughput drops more than 50 percent” or “events per second below N”, nothing fires. The failed pipeline can sit dead for hours while node-level graphs show a normal dip.

Logstash one input stopped: per-input failures masked by aggregate metrics

Mon, 01 Jan 0001 00:00:00 +0000

One of three inputs in your Logstash pipeline stops receiving events. The pipeline-level events.in counter drops by a third. But because the other two inputs keep flowing, the aggregate rate never hits zero, and your threshold-based alert stays silent. The failed input’s upstream source starts accumulating: Kafka consumer lag grows, file tails fall behind, or Beats agents buffer locally. By the time someone notices, hours or days of data from that source are delayed or lost.

Logstash OutOfMemoryError: Java heap space and how to recover

Mon, 01 Jan 0001 00:00:00 +0000

java.lang.OutOfMemoryError: Java heap space in the Logstash log means the JVM could not satisfy an allocation request and GC could not reclaim enough heap to proceed. The process exits or gets kernel OOM-killed, and every pipeline it was running stops with it.

Before the hard crash, there is usually a warning period: the GC death spiral. Heap fills, garbage collection runs longer and more frequently, throughput collapses. This can last minutes or hours before the JVM finally fails to allocate. If you catch the spiral, you can intervene before data is lost. If you only catch the OOM, you are in recovery mode.

Logstash output errors and retries: reading downstream failure before the queue grows

Mon, 01 Jan 0001 00:00:00 +0000

Output-side failures are the most common cause of Logstash queue growth and eventual outage. When the downstream destination (Elasticsearch, Kafka, an HTTP endpoint) starts rejecting, timing out, or slowing down, Logstash output plugins retry. Those retries preserve data temporarily but hide mounting delay. The pipeline appears alive while events accumulate.

The critical window for diagnosis is between the first output errors and the moment the queue fills. Once the queue reaches capacity, inputs are blocked, upstream systems buffer or drop events, and the incident has cascaded beyond Logstash.

Logstash persistent queue full: max_bytes reached and inputs blocked

Mon, 01 Jan 0001 00:00:00 +0000

The persistent queue has hit queue.max_bytes, input threads are blocked trying to push events in, and the pipeline has stopped accepting new data. A full PQ is almost never the root cause. It is the terminal stage of a downstream outage or capacity failure that the queue was absorbing. Fix the cause, drain the queue, prevent recurrence.

This guide covers confirming the state, finding the downstream cause, calculating how much time you have, and recovering without making things worse. For the underlying throttling metric, see Logstash flow.queue_backpressure: the input-throttling metric explained.

Logstash persistent queue not draining: page-release lag after downstream recovery

Mon, 01 Jan 0001 00:00:00 +0000

The downstream outage is over. Elasticsearch is green, output errors have stopped, and Logstash is delivering events again. But the persistent queue is shrinking far slower than it filled, disk I/O is pinned, and df shows the same used space it did an hour ago even though queue_size_in_bytes has clearly dropped.

Most of the time, nothing is stuck. A persistent queue (PQ) draining after recovery looks unhealthy from almost every angle: high disk I/O, reduced effective throughput, and disk usage that refuses to go down. This is expected behavior driven by how PQ pages are released. The failure modes you actually need to rule out are narrower: page or checkpoint corruption, poison events pinning pages, and disk I/O saturation becoming the new bottleneck.

Logstash persistent queue runway: how long until the PQ fills

Mon, 01 Jan 0001 00:00:00 +0000

A persistent queue changes the failure shape of a Logstash pipeline. When the output destination goes down, nothing looks broken for a while: inputs keep accepting events, throughput counters keep moving, the process stays green, and the queue absorbs the gap between arrival and delivery. The outage exists the whole time; the PQ just delays the moment anyone feels it.

Once queue_size_in_bytes reaches max_queue_size_in_bytes, inputs block and upstream systems (Beats agents, Kafka consumers, TCP senders) back up or drop data. The question that matters during a downstream impairment is not “is the queue growing?” but “how long until it is full at this rate?” This guide covers how to compute that number from the node stats API, how to read it against expected recovery time, and the pitfalls that produce misleading answers.

Logstash pipeline not started: a healthy JVM with a missing pipeline

Mon, 01 Jan 0001 00:00:00 +0000

Systemd says Logstash is active. The monitoring API on port 9600 returns 200. Heap looks fine, GC is quiet, and yet one of your data sources has gone silent at the destination. When you query /_node/stats/pipelines, the pipeline ID you expected is not there, or its event counters have not moved in hours.

This is a partial outage that process-level checks cannot see. Since Logstash 7.11, a pipeline that crashes no longer takes the JVM down with it: the process stays alive, serving the API, with fewer pipelines running than you configured. A failed initial config load, a failed reload that stopped the old pipeline without starting the new one, or a worker error that terminated a single pipeline all produce the same shape: healthy JVM, missing pipeline.

Logstash pipeline stalled: output rate at zero while the process looks alive

Mon, 01 Jan 0001 00:00:00 +0000

The Logstash JVM is running. systemctl status logstash says active. The monitoring API on port 9600 returns 200. Every liveness check you have is green. But events.out has not moved in twenty minutes, and data stopped arriving downstream at about the same time.

This is the “living dead” state: the process exists, but the pipeline is functionally dead. It is one of the most common ways Logstash fails in production, and it is invisible to any monitoring that only checks process existence. Output throughput, not liveness, is the health signal.

Logstash pipeline.workers and batch_size: tuning the worker pool

Mon, 01 Jan 0001 00:00:00 +0000

The worker pool is the most important throughput lever in Logstash. pipeline.workers sets how many threads process events through the filter and output stages. pipeline.batch.size sets how many events each worker handles per trip through the queue. Together, these define the maximum in-flight event count, the pipeline’s memory footprint, and how much CPU and downstream capacity it can consume.

Tuning is not about finding a universal optimum. It is about matching the worker pool to three variables: CPU cost per event (dominated by filters), average event size, and the downstream system’s ability to absorb batches efficiently. Getting this wrong produces one of three outcomes: CPU saturation with queue growth, heap exhaustion from too many in-flight events, or output-blocking cascades that look like high worker utilization but produce no useful throughput.

Logstash queue events count growing: reading the in-flight backlog

Mon, 01 Jan 0001 00:00:00 +0000

queue.events_count trending upward on the pipeline stats API means one thing: events are arriving faster than workers can process and deliver them. It is the primary backpressure indicator, and the earliest warning you get before inputs block and upstream systems start backing up or dropping data.

What the number alone does not tell you is how much trouble you are in. The same rising curve means different things for the memory queue versus a persistent queue, and the absolute value is workload-dependent enough that fixed thresholds are nearly useless. A memory queue climbing by a few hundred events can be more urgent than a persistent queue holding a million.

Logstash queue full: inputs blocked and the backpressure wedge

Mon, 01 Jan 0001 00:00:00 +0000

The symptom usually arrives from upstream first: Filebeat stops shipping, Kafka consumer lag climbs, or an HTTP input starts refusing connections. Logstash itself looks alive. The API answers on port 9600, the process is running, CPU is often unremarkable. But events are not moving, because the internal queue between inputs and workers is full, and every input thread is blocked trying to push into it.

This is the backpressure wedge: outputs slow or fail, workers cannot drain the queue, the queue fills, inputs block, and upstream systems back up or drop events. The queue is where the pain shows up, but it is almost never where the problem lives.

Logstash silent data loss: events dropped with no DLQ and green dashboards

Mon, 01 Jan 0001 00:00:00 +0000

The first sign is usually a question from downstream: “Where are last Wednesday’s logs?” Your Logstash dashboards show green. Process liveness is fine. Events-in and events-out curves track each other. The queue is empty. GC is quiet. Nothing paged.

Silent data loss happens when events leave the Logstash accounting system as “delivered” but never reach their destination in usable form. The events.out counter increments, throughput looks normal, and no alert fires. By the time anyone notices, the loss window may be hours or days old and the evidence is gone.

Logstash SSL/TLS handshake failures: expired certs and broken trust chains

Mon, 01 Jan 0001 00:00:00 +0000

Your Logstash logs are filling with PKIX path building failed or handshake_failure errors. Depending on which side is failing, you either have a delivery outage in progress (an output that can no longer connect to its destination) or a security signal (clients failing to authenticate against an input). The triage path starts the same either way: find the failing plugin, read the actual certificate error, and determine whether this is an availability problem, a security problem, or a planned rotation you forgot about.

Logstash thread starvation: workers blocked, low CPU, and stalled throughput

Mon, 01 Jan 0001 00:00:00 +0000

Logstash throughput has collapsed. The queue is growing. Your pipeline workers are all occupied. But CPU is sitting at 20% with no obvious explanation. Restarting the process clears it temporarily, then the pattern repeats. This is thread starvation. The symptoms look contradictory, which makes it easy to misdiagnose.

The mechanism: pipeline worker threads are stuck waiting on a shared resource rather than doing computation. Each worker pulls a batch from the queue and does not release it until every filter and output in the chain completes. When the thing that completes that chain is a slow network call, an exhausted connection pool, a mutex held by another worker, or a DNS resolver that hangs, the worker blocks. With enough workers blocked, no one drains the queue, backpressure propagates to inputs, and the pipeline stalls. CPU stays low because threads are parked in wait states, not burning cycles.

Logstash too many open files: file-descriptor exhaustion and refused connections

Mon, 01 Jan 0001 00:00:00 +0000

Logstash is logging java.io.IOException: Too many open files (EMFILE), Beats shippers report refused or reset connections, and new files are not being tailed. The JVM process is still alive, systemd still says active (running), and the monitoring API on port 9600 may still answer. Nothing looks crashed, yet the pipeline has stopped accepting new work.

That is the defining shape of file-descriptor exhaustion: a partial failure that is routinely misread as a network or disk problem. The process cannot open new sockets, new persistent queue page files, or new files for the file input, because every open of those consumes one file descriptor and the per-process limit has been reached. Existing connections and already-open files keep working, which is why some traffic continues to flow while new connections are refused.

Logstash won't start after a crash: persistent queue corruption and checkpoint errors

Mon, 01 Jan 0001 00:00:00 +0000

Logstash was killed uncleanly (OOM kill, kill -9, SIGKILL after a pod termination grace period, power loss) and now refuses to start. The log at /var/log/logstash/logstash-plain.log shows java.io.IOException and checkpoint-related errors during pipeline initialization, and the process exits before any events flow.

The persistent queue (PQ) is a page-based on-disk queue: events live in page files (default 250MB each, up to queue.max_bytes which defaults to 1GB), and checkpoint files record which events have been acknowledged as delivered. Both are updated continuously while the pipeline runs. An unclean shutdown can leave the checkpoint files and page files inconsistent with each other, and Logstash’s startup code refuses to open a queue it cannot reconcile.

Logstash worker utilization high: reading flow.worker_utilization and per-plugin skew

Mon, 01 Jan 0001 00:00:00 +0000

High flow.worker_utilization by itself means workers are busy. It becomes actionable only when the queue is growing or there is no headroom for traffic spikes. The first check is always queue depth, not utilization.

flow.worker_utilization is a percentage from 0 to 100, available in Logstash 8.x at the pipeline level. Plugin-level breakdowns live under plugins.filters[].flow.worker_utilization and plugins.outputs[].flow.worker_utilization in the Node Stats API. Pipeline-level tells you whether workers are saturated. Plugin-level tells you which filter or output is responsible.