$ guides / logstash / logstash-cpu-bound-filters-grok-hell ▌

Operations Guides

Logstash CPU-bound filters (grok hell): high CPU, saturated workers, growing queue

Worker utilization is pinned above 90%. Host CPU is near its ceiling on allocated cores. The queue is growing, per-event processing duration is climbing, and output errors are absent. The pipeline is not blocked downstream. It is starved for compute because one or more filter plugins cannot keep up with the event rate.

The usual culprit is complex grok regex evaluation, though Ruby filters, heavy JSON manipulation, and enrichment plugins can produce the same signature. The critical diagnostic fork: if CPU is high, you have a compute problem. If CPU is low or moderate with the same queue growth, you have backpressure or I/O blocking, and the fixes are entirely different. This article covers the compute-bound path.

What this means

In a healthy pipeline, worker threads pull batches from the queue, process them through the filter chain, hand them to outputs, and loop. When a filter is CPU-bound, each worker spends its time inside the filter instead of cycling through batches. The queue drains slowly or stops draining. Events accumulate. If the queue is memory-backed, inputs get backpressured quickly. If persistent queue is enabled, you have more runway, but the clock is ticking.

The defining characteristics, drawn from the pipeline stats API:

flow.worker_utilization sustained above 90% (Logstash 8.x flow metric)
Host CPU near 100% of allocated cores
Queue events_count trending upward
Per-event processing duration (duration_in_millis / filtered) rising
Output errors and retries absent or at baseline
One filter plugin dominates per-plugin worker_utilization or duration_in_millis

This pattern is distinct from downstream backpressure, where CPU is low because workers are waiting on output I/O, and from GC death spiral, where CPU is high but dominated by garbage collection rather than useful filter work.

Per-plugin flow.worker_utilization is exposed in the stats API for filter and output plugins since Logstash 8.6 (elastic/logstash#14743); pipeline-level flow metrics exist since 8.5, and pipeline-level worker_utilization was added in Logstash 8.13 (elastic/logstash#15912).

flowchart TD
    A["Queue growing, throughput dropping"] --> B{"Host CPU near ceiling?"}
    B -->|"No, low or moderate"| C["Backpressure: output blocked or I/O wait"]
    B -->|"Yes"| D{"GC overhead > 20% of wall time?"}
    D -->|"Yes"| E["GC death spiral: heap pressure"]
    D -->|"No"| F["Compute bottleneck: filter saturation"]
    F --> G["Capture hot_threads API snapshot"]
    G --> H{"Stack traces show regex or grok?"}
    H -->|"Yes"| I["Grok hell: optimize, anchor, or bypass"]
    H -->|"No"| J["Other CPU filter: ruby, json, dissect, geoip"]

Common causes

Cause	What it looks like	First thing to check
New log format triggers grok backtracking	Sudden CPU spike after a source application deploy, grok `failures` counter rising	Hot threads API for `org.joni` stack frames
Config rollout added filter complexity	CPU spike correlates with config reload timestamp	Per-plugin `duration_in_millis` for the new or changed filter
Traffic mix shift toward expensive events	Same config, same volume, but per-event duration up	Compare event size distribution and source breakdown before and after
JRuby Timeout overhead in grok	High CPU with moderate pattern complexity, throughput below expected for the hardware	Grok plugin version and `timeout_millis` setting
Ruby filter with inefficient code or allocation	Hot threads show Ruby execution, heap may climb alongside CPU	Per-plugin stats for the ruby filter, hot threads stack

Quick checks

Run these read-only API calls against the Logstash monitoring API. All are safe and non-disruptive.

# Pipeline-level: worker utilization, queue depth, throughput ratios
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty

# Per-plugin: which filter dominates duration and worker utilization
# Replace "main" if your pipeline has a different name
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | python3 -c "
import sys, json
filters = json.load(sys.stdin)['pipelines']['main']['plugins']['filters']
for f in filters:
    name = f.get('name', '?')
    fid = f.get('id', '?')
    flow = f.get('flow', {})
    events = f.get('events', {})
    print(f\"{name} ({fid}): wu={flow.get('worker_utilization',{})}, dur={events.get('duration_in_millis',0)}ms\")
"

# Hot threads: confirm grok or regex in stack traces
curl -sS 'http://127.0.0.1:9600/_node/hot_threads?threads=10&human=true'

# JVM GC stats: rule out GC death spiral
curl -sS http://127.0.0.1:9600/_node/stats/jvm?pretty | python3 -c "
import sys, json
gc = json.load(sys.stdin)['jvm']['gc']['collectors']
for gen, stats in gc.items():
    print(f\"{gen}: {stats['collection_count']} collections, {stats['collection_time_in_millis']}ms total\")
"

# Process CPU percentage
curl -sS http://127.0.0.1:9600/_node/stats/process?pretty | python3 -c "
import sys, json
p = json.load(sys.stdin)['process']
print(f\"CPU percent: {p.get('cpu',{}).get('percent','N/A')}\")"

# Output errors: confirm outputs are healthy (rule out backpressure)
# Adjust path for your installation method (RPM, DEB, tarball, container)
grep -Ei '(retry|error|exception|failed|reject|429|503)' /var/log/logstash/logstash-plain.log | tail -n 50

The goal: confirm high CPU, high worker utilization, growing queue, healthy outputs, and filter code in the hot threads stack. If any of these do not match, you are likely dealing with a different failure mode.

How to diagnose it

Confirm the compute bottleneck pattern. Check flow.worker_utilization and host CPU. Both should be high. Cross-reference with output error logs. If output errors are present and CPU is low, you have backpressure, not grok hell.
Capture hot threads. Run curl -sS 'http://127.0.0.1:9600/_node/hot_threads?threads=10&human=true' two or three times, 5 to 10 seconds apart. A single snapshot can be noisy. Look for stack frames containing org.joni.ByteCodeMachine.execute or org.jruby.RubyRegexp, which indicate regex evaluation inside the JRuby regex engine that grok uses. Worker threads burning 80% or more of their CPU in these frames confirm grok saturation.
Identify the dominant filter. Query per-plugin stats. Sort filters by duration_in_millis or flow.worker_utilization. The filter consuming a disproportionate share of processing time is the bottleneck. If you did not assign explicit id values to your filter plugins, the auto-generated IDs will be opaque. Fix this in your config so future diagnosis is faster.
Check the grok filter’s failures counter. A rising failures rate means patterns are not matching. Failed matches are significantly more expensive than successful matches because the regex engine exhausts all pattern alternatives before giving up. A format change at the source can turn a previously fast pipeline into a CPU hog overnight.
Check grok plugin version and timeout settings. The default timeout_millis is 30000, i.e. 30 seconds (the option is denominated in milliseconds; the plugin docs note the actual expiry is approximate, quantized to about 250 ms). The grok filter uses a timeout mechanism to abort pathological patterns, and this mechanism itself can become a source of thread contention under load: grok 4.1.0 introduced timeout enforcement through the JRuby Timeout class, which caused the throughput regression tracked in elastic/logstash#11302 (recommended workaround: timeout_millis => 0; grok 4.2.0 added timeout_scope => "event" to cut the per-pattern overhead).
Compare current event mix to last known good state. If nothing changed in config but CPU spiked, check whether a source application changed its log format. A new field, a different timestamp format, or extra whitespace can cause grok to try every pattern in the match block before failing.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`flow.worker_utilization`	Primary saturation indicator. Shows how close workers are to capacity.	Sustained above 90% with queue growth
`process.cpu.percent`	Confirms CPU is the bottleneck, not I/O wait or lock contention.	Near 100% of allocated cores, sustained
Queue `events_count` trend	Shows whether the pipeline is keeping up.	Monotonic increase over 10+ minutes
Per-event duration (`duration_in_millis / filtered`)	Rising per-event cost means filters are getting more expensive.	Greater than 2x baseline without config change
Per-plugin `worker_utilization`	Localizes the bottleneck to a specific filter.	One filter dominates the share
Grok `failures` counter rate	Failed matches are far more expensive than successful ones.	Sudden increase above baseline
GC overhead (`collection_time_in_millis` delta)	Rules out GC death spiral, which also shows high CPU.	GC time exceeds 20% of wall time
Output error/retry rate	Rules out downstream backpressure.	Sustained non-zero pattern

Fixes

Anchor and simplify grok patterns

Unanchored grok patterns force the regex engine to search for matches at every position in the string. Adding ^ and $ anchors tells the engine to match from the start and end of the line only, which can make failure detection roughly 10x faster because the engine rejects non-matching lines immediately instead of scanning.

Order patterns from most common to least common. With break_on_match set to true (the default), grok stops at the first successful match. If your most common log format is last in the match block, every common event pays the cost of failing through all preceding patterns.

Reduce the number of patterns in a single match block. If you have 20+ patterns, consider splitting by log type using conditionals on a field that identifies the source, so each event only passes through the relevant patterns.

Tune or disable grok timeout

The grok filter’s timeout mechanism can itself become a source of thread contention and throughput degradation under load.

timeout_scope is a real grok filter parameter (values pattern or event, default pattern), introduced in the grok filter 4.2.0 (logstash-plugins/logstash-filter-grok#153).

Two options:

Set timeout_millis => 0 to disable timeout enforcement entirely. This removes the contention but also removes the safety net against pathological patterns that could run indefinitely. Use this only if you have tested your patterns against your data and are confident they do not exhibit catastrophic backtracking.
Set timeout_scope => "event" to apply a single timeout across all patterns in a match block, rather than per-pattern. The official plugin docs recommend event-scoped timeouts as usually better. This helps with multi-pattern fallthrough scenarios, though its impact on single-pattern pipelines is limited.

Replace grok with dissect where format is stable

The dissect filter tokenizes fixed-format logs using delimiter-based splitting instead of regex. For logs with a reliable, repeating structure (syslog, access logs, many application log formats), dissect is significantly faster and uses less CPU. If your format has optional fields or variable structure, you may still need grok, but many pipelines can use dissect for the majority of events and reserve grok for the complex minority.

Increase pipeline.workers only if CPU headroom exists

The default pipeline.workers equals the number of CPU cores. Increasing it helps only if there is unused CPU capacity. If CPU is already at 100% on allocated cores, adding workers increases context switching without improving throughput. In containerized deployments, check whether CFS quota throttling is already limiting the process. More workers in a throttled container makes the problem worse.

Isolate or bypass the expensive filter

If you cannot fix the pattern immediately, reduce blast radius. Route the expensive event type to a separate pipeline with its own queue and worker pool, or temporarily drop or tag the offending event stream so the main pipeline can recover. This is a triage action, not a permanent fix.

Prevention

Anchor all grok patterns. Use ^ and $ unless you have a specific reason not to.
Assign explicit id values to every filter plugin. Auto-generated IDs make per-plugin stats impossible to map back to config during incidents.
Monitor per-plugin worker_utilization and duration_in_millis. Aggregate pipeline metrics hide a single dominant filter.
Set baseline-relative alerts on flow.worker_utilization. Sustained above 90% during normal production peaks means the pipeline cannot absorb traffic spikes without queue growth.
Watch the grok failures counter. A rising failure rate means patterns are not matching, and each failure is more expensive than a success. This is often the first sign of source format drift.
Test pattern changes against production samples. A pattern that works on 10 test events may exhibit catastrophic backtracking on the 11th. Run new patterns against a representative sample of real logs before deploying.
Consider dissect for structured logs. Reserve grok for formats that genuinely need regex. Fixed-delimiter logs parse faster and cheaper with dissect.

How Netdata helps

Correlate flow.worker_utilization with host CPU and queue depth in a single per-second view, making the compute bottleneck pattern immediately distinguishable from backpressure or GC pressure.
Per-plugin stats from the Logstash API can be collected and charted individually, surfacing which filter dominates processing time without manual curl and jq during an incident.
ML anomaly detection on per-event processing duration catches gradual degradation (format drift, growing pattern complexity) before it becomes a queue-growth incident.
Grok failures counter rate monitored as a trend signal provides early warning when source formats change and matches start failing.
GC overhead and heap metrics displayed alongside CPU utilization let you rule out GC death spiral in seconds, which is the most common misdiagnosis when both patterns show high CPU.