$ guides / logstash / logstash-pipeline-workers-batch-tuning ▌

Operations Guides

Logstash pipeline.workers and batch_size: tuning the worker pool

The worker pool is the most important throughput lever in Logstash. pipeline.workers sets how many threads process events through the filter and output stages. pipeline.batch.size sets how many events each worker handles per trip through the queue. Together, these define the maximum in-flight event count, the pipeline’s memory footprint, and how much CPU and downstream capacity it can consume.

Tuning is not about finding a universal optimum. It is about matching the worker pool to three variables: CPU cost per event (dominated by filters), average event size, and the downstream system’s ability to absorb batches efficiently. Getting this wrong produces one of three outcomes: CPU saturation with queue growth, heap exhaustion from too many in-flight events, or output-blocking cascades that look like high worker utilization but produce no useful throughput.

What the worker pool does

Each pipeline worker thread pulls a batch of events from the queue, runs every filter plugin in sequence on that batch, pushes the results to the output plugins, and waits for the output to acknowledge the batch before pulling the next one. Filters and outputs run in the same thread. There is no separate output worker pool.

The key implication: a worker blocked on output I/O is not processing events. If the Elasticsearch output takes 500ms to acknowledge a bulk request, that worker produces zero filter throughput for those 500ms. With pipeline.workers set to the host’s CPU core count (the default), a single slow output can stall a meaningful fraction of total processing capacity.

How workers, batches, and the queue interact

flowchart TD
    Q["Queue"] -->|"pull batch_size events"| W1["Worker 1"]
    Q -->|"pull batch_size events"| W2["Worker 2"]
    Q -->|"pull batch_size events"| WN["Worker N"]
    W1 --> S1["filters then outputs"]
    W2 --> S2["filters then outputs"]
    WN --> SN["filters then outputs"]
    S1 -.->|"output ack"| Q
    S2 -.->|"output ack"| Q
    SN -.->|"output ack"| Q

The three settings that govern this loop:

Setting	Default	What it controls
`pipeline.workers`	Number of host CPU cores	Thread count for the combined filter and output stage
`pipeline.batch.size`	125 events	Events each worker pulls from the queue per batch
`pipeline.batch.delay`	50ms	Maximum wait for a full batch before processing a partial one

A worker pulls up to batch.size events from the queue. If fewer than batch.size events are available, the worker waits up to batch.delay milliseconds for more to arrive. If the delay expires, the worker processes whatever partial batch it has. Once the batch is processed and the output acknowledges it, the worker pulls the next batch.

In-flight events and heap pressure

The maximum number of events held simultaneously by worker threads is:

in-flight events (max) = pipeline.workers * pipeline.batch.size

With defaults on an 8-core host: 8 workers times 125 events = 1,000 events in worker memory. Each event occupies JVM heap from the moment a worker pulls it until the output acknowledges the batch. Doubling workers or doubling batch.size doubles the in-flight count. Doubling both quadruples it.

With the default in-memory queue, events waiting in the queue also consume heap. With persistent queue (PQ) enabled, queued events live on disk, so heap pressure comes primarily from the worker-held events above.

If events average 100KB (large JSON payloads with nested fields), 1,000 in-flight events consume roughly 100MB of heap purely for worker buffering. Increasing workers from 8 to 16 without checking heap headroom can push the JVM into GC pressure that negates any throughput gain.

Before increasing workers or batch.size, verify that JVM heap has room for the multiplied in-flight count times the worst-case event size.

Multi-pipeline contention

In multi-pipeline deployments (using pipelines.yml), each pipeline gets its own worker pool. Workers from different pipelines share the same JVM heap and compete for the same CPU cores. If two pipelines each default to pipeline.workers equal to the host’s 16 cores, you have 32 worker threads competing for 16 cores. Total workers across all pipelines should not wildly exceed available cores.

Settings can be applied globally in logstash.yml or overridden per-pipeline in pipelines.yml. Verify which file is active:

# Check global settings
grep -E 'pipeline\.(workers|batch)' /etc/logstash/logstash.yml

# Check per-pipeline overrides
cat /etc/logstash/pipelines.yml

Where worker-pool tuning goes wrong

Container CPU detection

Logstash sets pipeline.workers to the number of CPU cores it detects at startup. In containers, this detection reads cgroup data, but on older JDKs it could report the host’s core count rather than the container’s CPU limit. On a 32-core host running a container with a 4-CPU limit, Logstash may start 32 workers competing for 4 cores worth of CPU time. The result is context-switch overhead, CFS throttling, and throughput worse than a 4-worker configuration.

Current Logstash 8.x with the bundled JDK (17/21) is cgroup-aware (cgroup v1 since JDK 10, cgroup v2 since JDK 15), which fixed the 2017-2018-era host-core misdetection (elastic/logstash#6797, #6824). Still, verify what the container actually lets you use:

Operators running Logstash in Docker or Kubernetes should explicitly set pipeline.workers to match the container’s CPU limit rather than relying on auto-detection. CFS throttling is visible in the cgroup’s cpu.stat file:

# Check for CFS throttling (path varies by cgroup version)
cat /sys/fs/cgroup/cpu.stat 2>/dev/null | grep -i throttled
cat /sys/fs/cgroup/cpu/cpu.stat 2>/dev/null | grep -i throttled

Non-zero nr_throttled or growing throttled_time means the container is hitting its CPU limit. Workers appear busy but are being throttled by the scheduler, not doing useful work.

Treating high worker utilization as a CPU problem

Worker utilization measures how busy workers are, not what they are busy doing. A pipeline with worker_utilization at 95% might have all workers pegged in grok regex evaluation (CPU-bound, where more workers or faster filters help), or all workers blocked waiting for Elasticsearch to acknowledge bulk requests (IO-bound, where adding workers only increases the number of threads waiting on the same slow output).

The distinguishing signal is CPU. If process.cpu.percent is high alongside high worker utilization, the pipeline is CPU-bound and more workers may help if CPU cores remain available. If CPU is low and worker utilization is high, workers are blocked on output I/O and adding workers will not improve throughput.

Increasing batch.size without checking downstream behavior

Larger batches can improve output efficiency because each bulk request to Elasticsearch (or batch send to Kafka) amortizes connection and serialization overhead over more events. But larger batches also:

Increase heap pressure per worker.
Increase per-batch output latency, since a 500-event batch takes longer to serialize and send than a 125-event batch.
Increase the risk that a single rejected event in a batch triggers a retry of the entire batch.

If the downstream system has bulk size limits or thread pool queue depth constraints, increasing batch.size beyond what the destination efficiently handles can produce bulk rejections that negate the batch efficiency gain. Check output plugin retry counts and error rates after any batch.size increase.

Tuning decisions: CPU-bound versus IO-bound

CPU-bound pipelines (complex grok patterns, Ruby filters, heavy JSON manipulation): Increase pipeline.workers incrementally as long as CPU headroom exists. Monitor process.cpu.percent and flow.worker_utilization. If adding a worker does not increase flow.output_throughput, the pipeline has hit the CPU ceiling and more workers will not help. The fix is filter optimization (simpler patterns, replacing grok with dissect where possible, caching enrichment lookups) or more CPU cores.

IO-bound pipelines (light filters, slow or rate-limited outputs): Increase pipeline.workers to add concurrency, so that while some workers wait on output I/O, others continue processing. Also consider increasing pipeline.batch.size to improve downstream batch efficiency. Monitor output plugin duration and retry counts to verify the downstream system is keeping up.

In both cases, tune one variable at a time. Change workers or batch.size, measure sustained output throughput and queue behavior over several minutes (not seconds), and keep the change only if it improves delivery rate without degrading heap or latency.

# Measure output throughput before and after a tuning change
curl -sS http://127.0.0.1:9600/_node/stats/pipelines/main | \
  python3 -c "import sys,json; d=json.load(sys.stdin)['pipelines']['main']; print('out:', d['events']['out'])"
# Wait 60 seconds, run again, compute events/sec from the delta

Deprecated settings to remove from old configs

If you encounter these in inherited configurations, they do nothing on modern Logstash:

pipeline.output.workers: Deprecated since Logstash 5.x. The combined worker pool (pipeline.workers) handles both filter and output execution. Remove it from your config.
Per-output workers option (for example, workers => 4 inside an elasticsearch output block): Also deprecated. Output plugins share the pipeline worker pool.

These settings do not cause errors on current Logstash versions, but they create false confidence that output concurrency is independently tunable. Since Logstash 5.0 output plugins are shared across the pipeline worker pool rather than owning per-plugin threads, which made per-output workers a no-op.

Signals to watch in production

Signal	Why it matters for worker-pool tuning	Warning sign
`flow.worker_utilization`	Direct measure of how close the pool is to saturation	Sustained above 90% with growing queue means throughput is capacity-limited
`flow.worker_concurrency`	How many workers are active on average (Logstash 8.6+)	Concurrency near `pipeline.workers` with flat throughput means workers are blocked, not processing
`process.cpu.percent`	Distinguishes CPU-bound from IO-bound worker saturation	High CPU and high utilization means CPU-bound; low CPU and high utilization means IO-bound
`flow.output_throughput`	Whether tuning changes actually improve delivery rate	No throughput increase after adding workers means the bottleneck was misidentified
`queue.events_count`	Whether the worker pool is keeping up with input	Growing queue after a tuning change means the change did not help
`jvm.mem.heap_used_percent`	Whether in-flight event count is creating memory pressure	Heap rising after increasing workers or batch.size means in-flight events consume too much memory
`plugins.filters[]` stats	Per-plugin breakdown of where worker time goes (`plugins.filters[].flow.worker_utilization` / `worker_millis_per_event` in the Node Stats API)	One filter dominating duration means filter optimization, not more workers, is the fix

Hot threads for confirmation

When worker behavior is ambiguous, hot threads reveals what workers are actually spending time on:

# Check what worker threads are doing right now
curl -sS 'http://127.0.0.1:9600/_node/hot_threads?pretty'

Stack traces in grok, regex, or Ruby code indicate CPU-bound filtering. Stack traces in HTTP client, socket read, or SSL handshake code indicate output blocking. Take multiple samples a few seconds apart to distinguish steady bottlenecks from transient work.

Correlating worker-pool signals in Netdata

The CPU-bound versus IO-bound distinction requires correlating flow.worker_utilization with process.cpu.percent at the same timestamp. Netdata collects both at per-second resolution, so the pattern (high/high versus high/low) is immediately visible without manual API polling or rate calculations between samples.

After any worker or batch.size change, watch these correlations:

jvm.mem.heap_used_percent rising alongside the increased in-flight count indicates GC pressure that may negate throughput gains.
queue.events_count flat or declining while flow.output_throughput rises confirms the change improved delivery. Queue depth rising while throughput stays flat means the bottleneck shifted downstream or was misidentified.
Per-plugin metrics isolate whether one filter dominates worker time, pointing to filter optimization rather than more workers.
Anomaly detection on flow.output_throughput and flow.worker_utilization surfaces regressions after a tuning change before queue growth makes them obvious.