$ guides / logstash / logstash-worker-utilization-high ▌

Operations Guides

Logstash worker utilization high: reading flow.worker_utilization and per-plugin skew

High flow.worker_utilization by itself means workers are busy. It becomes actionable only when the queue is growing or there is no headroom for traffic spikes. The first check is always queue depth, not utilization.

flow.worker_utilization is a percentage from 0 to 100, available in Logstash 8.x at the pipeline level. Plugin-level breakdowns live under plugins.filters[].flow.worker_utilization and plugins.outputs[].flow.worker_utilization in the Node Stats API. Pipeline-level tells you whether workers are saturated. Plugin-level tells you which filter or output is responsible.

The critical diagnostic fork: high worker utilization combined with high host CPU means filters are compute-bound. High worker utilization combined with low host CPU means workers are occupied but not doing useful compute. They are blocked on output I/O, lock contention, or a slow downstream dependency. Adding workers to the second case does not help and can make things worse.

What this means

flow.worker_utilization represents the percentage of pipeline worker thread time spent processing events. Conceptually it is duration_in_millis divided by uptime * pipeline.workers, scaled to 0-100. Pipeline workers (configured by pipeline.workers, defaulting to the CPU core count) pull batches from the queue, process them through the filter chain, and push to outputs. A worker counts as “utilized” during all of that work, including time spent blocked waiting for output acknowledgment.

This is the key subtlety. A worker blocked on a slow Elasticsearch bulk response is still “utilized.” It cannot pull the next batch, but it is not burning CPU. The correlation between worker utilization and host CPU is the single most important diagnostic signal when utilization is high.

When high utilization is fine:

Queue stays flat: events drain as fast as they arrive.
End-to-end latency is within acceptable bounds.
No output errors or retry storms in the logs.

When high utilization is a warning:

Sustained above 90% during normal production peaks: no headroom for traffic spikes. A burst will immediately start growing the queue.
Queue is already growing, even slowly. The pipeline has crossed from “efficient” to “falling behind.”
Output errors or retries are present. Workers are blocked, not computing.

When high utilization is misleading:

Worker utilization near 100% with host CPU well below the core ceiling. Workers are occupied but not doing compute. The bottleneck is I/O, lock contention, or downstream latency, not processing capacity.

flowchart TD
    A["worker_utilization > 90%"] --> B{"Host CPU near limit?"}
    B -->|"Yes, CPU saturated"| C{"Queue growing?"}
    B -->|"No, CPU has headroom"| D{"Output errors or retries?"}
    C -->|"Yes"| E["Compute bottleneck
Check per-plugin skew"]
    C -->|"No"| F["No headroom for spikes
but currently stable"]
    D -->|"Yes"| G["Output I/O blocking
Investigate downstream"]
    D -->|"No"| H["Lock contention or
non-CPU blocking"]

Common causes

Cause	What it looks like	First thing to check
CPU-bound filter saturation	High utilization + high CPU + growing queue. No output errors. Per-plugin stats show one filter dominating.	`plugins.filters[].flow.worker_utilization` for the expensive filter
Output I/O blocking	High utilization + low CPU + growing queue. Output errors, retries, or 429s in logs.	Output plugin duration and error counts
Lock contention	High utilization + low CPU + growing queue. No output errors. Hot threads show BLOCKED or TIMED_WAITING states.	`/_node/hot_threads` for thread state
Insufficient workers	High utilization + growing queue + CPU has headroom. All workers occupied but CPU is not the ceiling.	`pipeline.workers` vs available CPU cores
Overprovisioned workers with PQ	High utilization but EPS degrades. CPU well below capacity despite all workers busy.	Reduce `pipeline.workers` toward core count and compare throughput

Quick checks

All commands are read-only and safe to run during production.

# Read pipeline-level worker utilization and queue depth
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | python3 -c "
import sys, json
data = json.load(sys.stdin)
for pid, p in data.get('pipelines', {}).items():
    flow = p.get('flow', {})
    q = p.get('queue', {})
    print(f'{pid}: worker_utilization={flow.get(\"worker_utilization\",\"?\")} queue_events={q.get(\"events_count\",0)}')
"

# Per-plugin worker_utilization and worker_millis_per_event for filters and outputs
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | python3 -c "
import sys, json
data = json.load(sys.stdin)
for pid, p in data.get('pipelines', {}).items():
    for ptype in ('filters', 'outputs'):
        for plugin in p.get('plugins', {}).get(ptype, []):
            flow = plugin.get('flow', {})
            print(f'{pid}/{ptype}/{plugin.get(\"id\",\"?\")}: {flow}')
"

# Check host CPU for the Logstash process
curl -sS http://127.0.0.1:9600/_node/stats/process?pretty | python3 -c "
import sys, json
cpu = json.load(sys.stdin)['process']['cpu']
print(f'CPU percent: {cpu.get(\"percent\",\"?\")}')
"

# Capture hot threads to see what workers are actually doing
curl -sS 'http://127.0.0.1:9600/_node/hot_threads?pretty'

# Check for output errors and retry patterns in the log
grep -Ei '(retry|error|exception|failed|reject|429|503|timeout)' /var/log/logstash/logstash-plain.log | tail -n 200

# Check GC overhead (old-gen collections are the dangerous ones)
curl -sS http://127.0.0.1:9600/_node/stats/jvm?pretty | python3 -c "
import sys, json
gc = json.load(sys.stdin)['jvm']['gc']['collectors']
for gen, s in gc.items():
    print(f'{gen}: {s[\"collection_count\"]} collections, {s[\"collection_time_in_millis\"]}ms total')
"

# Check configured worker count and JVM uptime
grep pipeline.workers /etc/logstash/logstash.yml
curl -sS http://127.0.0.1:9600/_node/stats/jvm?pretty | grep uptime_in_millis

How to diagnose it

Gate on uptime. JVM uptime must be above 300 seconds (5 minutes) before the metric is trustworthy. During cold start, JIT compilation and grok pattern compilation inflate worker utilization. A 95% reading at 90 seconds of uptime is warmup noise.
Correlate utilization with host CPU. This is the diagnostic fork shown in the diagram above. process.cpu.percent from the API may report values above 100% on multi-core systems, as it is expressed as a per-core percentage.
- High utilization + CPU near the core ceiling: compute-bound. Proceed to step 3.
- High utilization + CPU well below the core ceiling: workers are occupied but not computing. Proceed to step 4.
If compute-bound, find the expensive plugin. Examine per-plugin worker_utilization and worker_millis_per_event for filters and outputs. One plugin consuming more than roughly 80% of pipeline processing time is the bottleneck. worker_millis_per_event shows the per-event cost for that plugin. If it reports Infinity, the plugin is spending worker time without completing any events: a stuck plugin or empty batches.
If not compute-bound, check output health. Look for output errors, retries, 429 responses, and timeouts in the log file and in per-output plugin stats. If output errors are present, the downstream destination is the root cause. Workers are blocked waiting for output acknowledgment.
If neither CPU nor output errors explain it, check for lock contention. Capture /_node/hot_threads and look for threads in BLOCKED or TIMED_WAITING state. This indicates workers are waiting on a shared lock or a non-CPU resource such as a DNS lookup or a mutex inside a non-thread-safe plugin.
Check whether workers are overprovisioned relative to CPU. If pipeline.workers is set well above the CPU core count and you are using persistent queues, throughput may degrade from write-thread contention. Workers compete for the PQ write lock, and effective utilization stays high while CPU remains moderate. Reducing workers toward the core count can improve throughput.
Check the queue trend. Regardless of root cause, a growing queue means the pipeline is falling behind. Calculate the fill rate and estimate runway: (max_queue_size - queue_size) / growth_rate.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`flow.worker_utilization`	Core saturation indicator. Shows how close workers are to their processing limit.	Sustained above 90% during normal peaks
Host CPU (`process.cpu.percent`)	Distinguishes compute-bound from I/O-bound saturation.	High util + low CPU means blocking, not compute
`queue.events_count`	Shows whether the pipeline is keeping up.	Monotonic growth over 15 minutes
`plugins.filters[].flow.worker_utilization`	Localises the bottleneck to a specific filter.	One filter above roughly 80% of pipeline time
`plugins.outputs[].flow.worker_utilization`	Localises the bottleneck to a specific output.	One output dominating processing time
`plugins.filters[].flow.worker_millis_per_event`	Per-event cost for each filter. Shows if a filter got more expensive over time.	Sudden increase, or `Infinity`
`flow.queue_backpressure`	How much input capacity is lost to queue pressure.	Rising significantly above baseline
Hot threads (`/_node/hot_threads`)	What workers are actually doing at the thread level.	BLOCKED or TIMED_WAITING in filter or output code
GC overhead (`jvm.gc.collectors.old`)	Whether GC is stealing worker time.	Old-gen GC above 10% of wall time

Fixes

CPU-bound filter

Optimize the expensive filter first. If a grok filter is the bottleneck, audit its patterns for catastrophic backtracking. A pattern that performs fine on well-formed input can exhibit exponential time on malformed input. Test patterns against real traffic samples, not synthetic data.

Increase pipeline.workers only if CPU headroom exists. If CPU is already near the core ceiling, more workers add context-switching overhead without throughput gain. Do not increase workers beyond available CPU cores for CPU-bound pipelines; threads compete for the same cores and throughput degrades from scheduling overhead.

Output I/O blocking

Fix the downstream issue. If Elasticsearch is rejecting bulk requests (429) or cluster health is yellow or red, Logstash workers will block regardless of worker count. The root cause is downstream, not in Logstash configuration.

Increase pipeline.workers for I/O-bound pipelines. Unlike CPU-bound pipelines, I/O-bound pipelines benefit from more workers than cores because workers can overlap I/O waits. While one worker waits for a bulk response, another can process the next batch.

Consider pipeline isolation. If one output is slow, move it to its own pipeline. This prevents a slow output from monopolizing workers that could be serving other outputs.

Lock contention

Identify the contended lock. Hot threads will show which code path threads are blocked on. Common causes: non-thread-safe filter plugins with internal state, DNS lookups without timeouts, or HTTP enrichment calls that block on a slow external service.

Split into multiple pipelines. If a single pipeline has high worker contention, splitting it into multiple pipelines with separate worker pools reduces lock contention. Each pipeline gets its own workers and queue.

Insufficient workers

Increase pipeline.workers up to the CPU core count. If utilization is high, the queue is growing, and CPU has headroom, the pipeline is underprovisioned.

For I/O-bound pipelines, going slightly above core count can help. Workers that spend most of their time waiting on output I/O can overlap waits, so a small number of extra workers improves throughput without saturating CPU.

Prevention

Track the utilization trend alongside CPU. A gradual rise in flow.worker_utilization with stable CPU over weeks indicates events are getting more expensive to process: data format drift, a more complex event mix, or filter configuration bloat. Catch this before utilization crosses 90%.

Monitor per-plugin worker_utilization routinely. One filter slowly increasing its share of processing time is the earliest signal of a future bottleneck. It is visible in per-plugin stats long before aggregate utilization saturates.

Size workers deliberately. Match pipeline.workers to the pipeline’s bottleneck type: CPU core count for compute-bound, slightly above for I/O-bound. Document the rationale so future operators do not change it without understanding the tradeoff.

Establish a utilization baseline for each pipeline during normal peaks. Alert on sustained deviation from that baseline, not on an arbitrary absolute threshold. A fixed “alert above 85%” rule fires constantly during legitimate peak hours and stays silent during a genuine degradation at a quieter period.

How Netdata helps

Per-second collection of flow.worker_utilization alongside host CPU metrics shows the compute-bound vs I/O-bound correlation without manual polling or computing deltas between API snapshots.
Per-plugin breakdown surfaces which filter or output is dominating before aggregate utilization hits 90%, giving lead time to optimize or isolate the slow plugin.
Anomaly detection on worker utilization baselines catches gradual degradation (a filter slowly getting more expensive as data format drifts) that static thresholds miss.
Correlating worker utilization with queue depth, output error rates, and GC overhead in a single view shortens the loop from “something is slow” to “this specific filter is the bottleneck and the queue is growing because of it.”