Logstash thread starvation: workers blocked, low CPU, and stalled throughput

Logstash throughput has collapsed. The queue is growing. Your pipeline workers are all occupied. But CPU is sitting at 20% with no obvious explanation. Restarting the process clears it temporarily, then the pattern repeats. This is thread starvation. The symptoms look contradictory, which makes it easy to misdiagnose.

The mechanism: pipeline worker threads are stuck waiting on a shared resource rather than doing computation. Each worker pulls a batch from the queue and does not release it until every filter and output in the chain completes. When the thing that completes that chain is a slow network call, an exhausted connection pool, a mutex held by another worker, or a DNS resolver that hangs, the worker blocks. With enough workers blocked, no one drains the queue, backpressure propagates to inputs, and the pipeline stalls. CPU stays low because threads are parked in wait states, not burning cycles.

The defining diagnostic signature is high worker concurrency combined with low CPU. This is the mirror image of CPU-bound filter saturation, where high concurrency comes with high CPU. If you conflate the two, you will add workers or increase batch sizes and make the problem worse.

What this means

Each worker thread owns a batch of events from the moment it pulls them from the queue until the output acknowledges delivery. The worker runs the entire filter chain sequentially, then hands the batch to output plugins. A worker blocks until the output call returns. If the output call does not return, the worker is stuck indefinitely with its batch.

When all workers are in this state, no batches leave the queue. The queue fills. Input threads eventually cannot push new events because the queue has no capacity. Backpressure reaches upstream systems: Beats agents buffer locally, Kafka consumer lag grows, TCP connections pile up. From outside, Logstash appears alive and running but is doing zero useful work.

Thread starvation is distinct from three other failure patterns that produce similar throughput collapse:

• CPU-bound filter saturation (grok hell): Workers are genuinely computing, CPU is maxed. Adding workers or CPU helps. See CPU-bound filters.
• GC death spiral: Heap is exhausted, GC consumes the JVM. CPU may be high but is spent on collection, not event processing. API becomes slow or unresponsive. Post-GC heap floor is rising.
• Output bottleneck cascade: Downstream rejects events (HTTP 429, connection refused), output retries dominate. Workers block on output acknowledgment, which looks like thread starvation. The distinction is that output errors and retries are visible in logs and plugin stats, while pure thread starvation from connection pool exhaustion or mutex contention may produce no errors at all.

flowchart TD
    A["Throughput drops,\nqueue grows"] --> B{"Host CPU\nnear capacity?"}
    B -->|"Yes"| C["Compute bottleneck\nSee: grok hell guide"]
    B -->|"No"| D["Thread starvation"]
    D --> E["Hot threads shows\nworkers BLOCKED or\nTIMED_WAITING"]
    E --> F["Identify blocking resource"]
    F --> G["Output I/O wait"]
    F --> H["DNS resolution"]
    F --> I["External HTTP call"]
    F --> J["Plugin lock contention"]

The CPU gate eliminates compute-bound patterns. Hot threads then identifies the specific blocking resource.

Common causes

Quick checks

Run these read-only commands during the incident. They are safe and non-disruptive.

# Check worker utilization and concurrency (Logstash 8.x flow metrics)
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -A5 '"flow"'

# Capture hot threads - the single most valuable diagnostic
curl -sS 'http://127.0.0.1:9600/_node/hot_threads?pretty'

# Check process CPU - should be low in thread starvation
curl -sS http://127.0.0.1:9600/_node/stats/process?pretty | grep -A3 '"cpu"'

# Check queue depth and growth
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -A10 '"queue"'

# Check output plugin duration per event
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -A20 '"outputs"'

# Check per-plugin worker cost (Logstash 8.x)
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -A5 '"worker_millis_per_event"'

# Check for output errors and retries in logs
grep -Ei '(retry|error|exception|failed|reject|unavailable|timeout|429|503)' /var/log/logstash/logstash-plain.log | tail -n 200

# Take a second hot threads sample 10 seconds later to confirm persistence
sleep 10 && curl -sS 'http://127.0.0.1:9600/_node/hot_threads?pretty'

The second hot threads sample is critical. A single snapshot can capture transient work. If workers show the same BLOCKED or TIMED_WAITING state on the same code path across two samples taken 10 seconds apart, the stall is persistent, not a momentary wait.

How to diagnose it

1. Confirm the signature. Check flow.worker_concurrency (should be at or near pipeline.workers) and process.cpu.percent (should be well below pipeline.workers * 100%). If CPU is high, you are looking at compute saturation, not thread starvation. Redirect to the grok hell guide.

2. Capture hot threads. Run _node/hot_threads and look at thread state for pipeline worker threads. The thread names typically follow the pattern [pipeline_name]>workerN or similar. Look for:

   • BLOCKED state: the worker is waiting on a lock held by another thread.
   • TIMED_WAITING on a socket read or HTTP client: the worker is waiting on network I/O.
   • WAITING on a synchronized monitor: possible plugin internal lock.

3. Identify the blocking code path. The stack trace in hot threads shows where the worker is parked. Common signatures:

   • java.net.SocketInputStream.socketRead0: network I/O wait (output or HTTP filter).
   • org.jruby or Resolv::DNS: DNS resolution blocking.
   • Object.wait or ReentrantLock: lock contention in a plugin.
   • Output plugin HTTP client classes (Manticore, Apache HttpClient): downstream slowness.

4. Check per-plugin stats. Look at plugins.filters[].flow.worker_millis_per_event and plugins.outputs[].events.duration_in_millis. The plugin with disproportionately high values is the bottleneck. In thread starvation, the blocking plugin shows high duration but the actual work (CPU) is minimal because the time is spent waiting, not computing.

5. Correlate with downstream health. If hot threads point at output I/O, check the destination independently. For Elasticsearch: _cluster/health, _nodes/stats for bulk rejections. For Kafka: broker status and consumer lag. For HTTP endpoints: direct connectivity test. The destination may be the root cause even though Logstash is where the symptom manifests.

6. Distinguish pool exhaustion from destination slowness. If the destination responds quickly to direct requests but Logstash workers are still stuck, the bottleneck is likely the connection pool size, not the destination itself. If the destination is slow under load, increasing the pool size will not help and may overwhelm the destination further.

Metrics and signals to monitor

Flow metrics (worker_utilization, worker_concurrency, queue_backpressure) are available since Logstash 8.6.0 (pipeline-level worker_utilization since 8.13.0); in earlier versions compute rates manually from cumulative counters.

Fixes

Output connection pool exhaustion

A worker blocks until the output call returns. If all connections in the output plugin’s pool are in use and none are being released (because the destination is slow), remaining workers queue behind the pool.

First, determine whether the destination is the problem or the pool size is. Test the destination directly under load. If it handles requests fast enough, the pool is undersized. If it is slow or saturated, fix the destination first.

If the destination is healthy, increase the connection pool size for the output plugin. The Elasticsearch output in Logstash 5.x+ uses a shared concurrency model that manages its own internal connection pool. For other outputs, check the plugin documentation for pool or connection settings.

Do not increase pipeline.workers to solve output blocking. More workers sending to a saturated or pool-limited output increases contention without improving throughput. More workers help only until CPU cores, lock contention, or downstream blocking dominates.

DNS lookup blocking

The dns filter performs DNS resolution synchronously in the worker thread. Ruby’s default resolver (Resolv::DNS) can block the thread for the full resolver timeout when the DNS server is slow or unreachable. Under certain conditions involving ndots in /etc/resolv.conf, lookups can hang until the system timeout.

Fixes:

• Add an explicit timeout to the dns filter configuration.
• Deploy a local caching resolver (dnsmasq, systemd-resolved) on the Logstash host to absorb repeated lookups and provide fast cache hits.
• Replace inline DNS enrichment with a pre-populated translate filter or dictionary lookup if the hostname set is bounded and known.

Plugin mutex contention

Some plugins are not thread-safe and hold internal locks. With high pipeline.workers, multiple workers contend on the same lock. Workers show BLOCKED state on the same synchronized block. The pattern worsens as you add workers, which is the opposite of what you would expect from a throughput problem.

Fixes:

• Split the pipeline into multiple pipelines, each with its own worker pool and its own plugin instance. This reduces per-pipeline worker contention. Note that pipelines still share the JVM heap, so this helps thread contention but not memory pressure.
• Reduce pipeline.workers for the affected pipeline as a temporary measure. Fewer workers means less lock contention, at the cost of throughput.
• Identify whether the plugin has a known thread-safety issue and check for updates or patches.

External HTTP call timeout

The http filter and ruby filter can make outbound HTTP calls. Without explicit timeouts, these calls block the worker thread until the default TCP timeout fires, which can be minutes on some platforms.

Fixes:

• Add explicit connect and read timeouts to all http filter and ruby filter external calls.
• Evaluate whether inline enrichment is necessary. If the external service is slow or unreliable, consider moving enrichment to a separate async pipeline or pre-computing lookup tables.
• Monitor the downstream service directly. If it is degraded, Logstash workers will block on every call regardless of timeout settings.

Prevention

Set explicit timeouts on every external call. DNS lookups, http filter calls, ruby filter HTTP calls, and any output with configurable timeouts. Default TCP timeouts are too long for production pipelines. A worker stuck on a 60-second timeout is a worker unavailable for useful work.

Monitor the worker utilization-to-CPU ratio. Under normal load, high worker utilization should correlate with high CPU. When worker utilization is high but CPU is low, something is blocking. This ratio is the earliest leading indicator of thread starvation before the queue starts growing.

Use per-plugin stats proactively. The worker_millis_per_event metric per plugin reveals which filter or output is getting slower over time. Catch a degrading plugin before it monopolizes all workers.

Isolate workload types into separate pipelines. Pipelines with different downstream targets, different enrichment requirements, or different reliability needs should not share a worker pool. A slow output in one pipeline should not starve workers in another. Multi-pipeline deployments (via pipelines.yml) give each pipeline its own queue and worker pool while sharing the JVM.

Validate downstream capacity before scaling workers. Adding workers increases concurrent output load. If the destination cannot handle more concurrent connections or requests, additional workers will block on output I/O and make the problem worse.

How Netdata helps

Netdata collects the metrics that distinguish thread starvation from other throughput collapses at per-second resolution:

• Worker utilization vs CPU correlation: worker utilization staying high while CPU drops signals blocking I/O before the queue fills.
• Queue depth trend: monotonic queue growth with low CPU is the thread starvation signature, visible without manual counter math.
• Output throughput vs input throughput divergence: the gap between these two rates quantifies how fast the backlog is building and how much runway remains.
• GC overhead alongside CPU: separates thread starvation (low CPU, low GC) from GC death spiral (high GC overhead, degrading API responsiveness).

• Hot threads capture: Netdata can trigger hot threads collection when throughput drops, providing stack traces at the moment workers stall.

Related guides

• Logstash CPU-bound filters (grok hell): high CPU, saturated workers, growing queue
• Logstash API unreachable on port 9600: crash, GC pause, or startup
• Logstash Beats input: Filebeat backpressure and connection health
• Logstash disk full: PQ, DLQ, and log volumes competing for space
• Logstash file descriptor pressure: leaks, tailed files, and reconnection churn