Logstash queue events count growing: reading the in-flight backlog

queue.events_count trending upward on the pipeline stats API means one thing: events are arriving faster than workers can process and deliver them. It is the primary backpressure indicator, and the earliest warning you get before inputs block and upstream systems start backing up or dropping data.

What the number alone does not tell you is how much trouble you are in. The same rising curve means different things for the memory queue versus a persistent queue, and the absolute value is workload-dependent enough that fixed thresholds are nearly useless. A memory queue climbing by a few hundred events can be more urgent than a persistent queue holding a million.

This article covers how to read the metric for each queue type, how to separate real growth from burst noise, and the three correlation patterns that tell you whether the backlog comes from expensive filters, a sick downstream, or blocked output I/O. For the broader signal taxonomy, see the Logstash monitoring checklist.

What queue.events_count actually counts

The metric comes from the node stats API:

# Queue state per pipeline
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty

Look at pipelines.<name>.queue. The fields you get, and how to interpret them, depend on queue.type.

Memory queue (default). A bounded in-memory buffer with no durability. events_count represents in-flight events, and the ceiling is small: roughly pipeline.workers (defaults to CPU core count) times pipeline.batch.size (default 125), so hundreds to a few thousand events. Two consequences: the queue fills fast and blocks inputs immediately when it does, and any monotonic increase over 15 minutes is concerning because there is no meaningful buffer absorbing the imbalance. Memory queue capacity is not directly exposed as a metric, so the trend of events_count and the input-side push duration are what you have.

Persistent queue. A page-based on-disk queue (default 64MB pages, default 1GB max_bytes). Here events_count can legitimately grow large during a downstream outage; that is what the PQ is for. Read it with the byte-level fields: queue_size_in_bytes against max_queue_size_in_bytes gives occupancy, and queue.data.free_space_in_bytes tells you how much filesystem runway the queue directory has. A PQ growing steadily while everything else looks healthy is a time bomb with a known fuse length, not a healthy system.

In both cases the absolute number is not the signal. The trend, specifically the rate of change, is.

Reading the curve: growth versus noise

Before treating a rising count as an incident, rule out the shapes that are normal:

• Spike that drains. Short accumulation during a traffic burst that returns to baseline between bursts is healthy buffering. This is what the queue exists for.
• Monotonic rise over 15+ minutes. The pipeline is falling behind. This is the pattern that matters.
• Oscillation between near-empty and near-full. Bursty input combined with borderline capacity. Not an incident yet, but fragile: any additional load or a slightly slower downstream tips it into sustained growth.
• Cold start buildup. In the first minutes after a restart, JIT warmup and plugin initialization slow workers, and a PQ replays events from the previous session. Gate any queue alarm on jvm.uptime_in_millis greater than about 300 seconds, and check whether a post-restart PQ is draining (occupancy declining) rather than filling.

One inversion worth remembering: a queue sitting at zero is not automatically healthy. If input throughput is also near zero when sources should be active, an empty queue means a dead input, not a healthy pipeline. Cross-reference with events.in or flow.input_throughput.

To quantify growth, sample events_count (memory) or queue_size_in_bytes (PQ) twice, 60 seconds apart, and compute the delta. On recent Logstash versions the PQ exposes this directly as flow.queue_persisted_growth_bytes: positive means filling, negative means draining. Flow metric names and availability vary between versions, so verify against your release.

The three correlation patterns

A growing queue has three common root causes, separable with two correlations: host CPU, and output error activity.

flowchart TD
  A[queue.events_count rising over 15+ min] --> B{Host CPU high?}
  B -- yes --> C[Compute bottleneck: filters are the cost]
  B -- no --> D{Output errors or retries rising?}
  D -- yes --> E[Downstream problem: destination slow or rejecting]
  D -- no --> F[Blocking I/O: workers waiting on output]
  C --> G[Confirm: per-plugin duration, hot threads in filter code]
  E --> H[Confirm: output plugin stats, destination health, log errors]
  F --> H

Queue grows plus CPU high: compute bottleneck. Workers are CPU-bound in the filter chain, typically grok patterns with pathological backtracking, heavy JSON manipulation, ruby filters, or enrichment lookups. Corroborate with flow.worker_utilization pinned high, rising per-event processing duration, one filter dominating plugins.filters[].events.duration_in_millis, and hot threads showing filter code. Output health looks normal. In containers, check CFS quota throttling first: the process can look moderately busy while actually being throttled.

Queue grows plus output errors rising: downstream problem. The destination is slow, rejecting, unavailable, or timing out. Corroborate with retry and error lines in logstash-plain.log (429s, timeouts, connection failures), per-output plugin stats, flow.output_throughput sitting below flow.input_throughput, and rising per-output duration. CPU is usually moderate here because workers wait on I/O rather than compute. With a PQ, this pattern can be masked for hours while the queue absorbs the gap.

Queue grows plus CPU low plus no output errors: blocking I/O on output. Workers are occupied but stuck waiting: connection pool exhaustion, network timeouts, or slow handshakes that have not surfaced as explicit errors yet. The tell is high worker utilization combined with low host CPU, and hot threads showing output wait paths rather than filter burn. This pattern is easy to misread as “nothing is wrong” because both the CPU graph and the error log look calm.

Quick checks

All read-only. Do not poll the stats API faster than about every 10 seconds on loaded instances.

# Queue type, depth, occupancy, and flow rates
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty

# JVM uptime (cold-start gate) and heap/GC pressure
curl -sS http://127.0.0.1:9600/_node/stats/jvm?pretty

# Process CPU
curl -sS http://127.0.0.1:9600/_node/stats/process?pretty

# What worker threads are doing right now (take 2-3 samples)
curl -sS 'http://127.0.0.1:9600/_node/hot_threads?pretty'

# Output-side errors and retries
grep -Ei '(retry|error|exception|failed|reject|unavailable|timeout|429|503)' /var/log/logstash/logstash-plain.log | tail -n 200

# Filesystem runway for PQ, DLQ, and logs
df -h /var/lib/logstash /var/log/logstash

In multi-pipeline deployments, aggregate numbers hide localized failure. Query /_node/stats/pipelines/<pipeline_id> per pipeline; one stalled pipeline out of five only moves the aggregate by 20 percent.

How to diagnose it

1. Confirm the queue type and the trend. Take two samples 60 seconds apart. For PQ, compute occupancy as queue_size_in_bytes / max_queue_size_in_bytes and note queue.data.free_space_in_bytes.
2. Gate on uptime. If jvm.uptime_in_millis is under about 300 seconds, wait and re-check. A PQ draining after a restart should show declining occupancy.
3. Compare input to output. If flow.output_throughput sits below flow.input_throughput on a smoothed window, the backlog is real and growing at the difference.
4. Split by CPU. High host CPU plus high flow.worker_utilization points at the filter chain. Low CPU with high utilization points at output waits.
5. Check output errors. Grep the log for retries, rejects, timeouts, and 429/503s, and look at per-output plugin stats. Errors plus growth means downstream; no errors plus low CPU means blocking I/O.
6. Use hot threads if still ambiguous. Filter code in the hot threads output confirms compute; output wait paths confirm I/O blocking. Take repeated snapshots; a single one is noisy.
7. Estimate runway (PQ). (max_queue_size_in_bytes - queue_size_in_bytes) / current fill rate gives time until inputs block. Under roughly 30 minutes of runway with active input and corroborated downstream impairment, treat it as a page, not a ticket.

Metrics and signals to monitor

Alert on rate of change and deviation from baseline, not absolute values. Absolute thresholds fire all night on low-traffic pipelines and never fire on growing ones.

Fixes by cause

Compute bottleneck. Identify the expensive stage with per-plugin duration, then simplify, bypass, or isolate it. Adding workers only helps if CPU headroom exists; on saturated cores it adds context switching and makes things marginally worse. If the trigger was a config rollout or a new log format, treat the filter change as the suspect.

Downstream problem. Fix the destination first; Logstash heals itself by draining the queue once delivery recovers, though PQ drain can take much longer than the fill did. While the downstream is down, watch runway: if the fill rate says the PQ will be full before recovery, reduce ingest or shed non-critical pipelines early instead of waiting for inputs to block.

Blocking I/O. Confirm wait paths with hot threads, then look at the output’s connection behavior and network path. For genuinely I/O-bound pipelines, raising pipeline.workers beyond core count can help because workers spend their time waiting rather than computing; the mental model in How Logstash actually works in production covers why a blocked worker stalls queue consumption.

PQ sizing. max_bytes must be sized relative to the filesystem, not just the workload: if the queue shares a partition with logs or the DLQ, the disk can fill before the queue limit is reached. Keep PQ max well under available filesystem space, and remember pages are released in 64MB jumps, not smoothly, so disk usage lags behind drain.

Do not restart Logstash as a first response to queue growth. A restart discards everything in a memory queue and, on a PQ, only postpones the question of why the backlog formed. The exception is a confirmed GC death spiral, which is a different failure pattern with its own signals. If growth has already crossed into a full queue with blocked inputs, the triage changes: see Logstash queue full: inputs blocked and the backpressure wedge.

Prevention

• Trend alerts, not level alerts. Page on smoothed growth over 5 to 15 minutes combined with output below input; ticket on 15 minutes of steady rise.
• Occupancy tiers for PQ. Warn at 80 percent sustained, page at 90 percent with positive growth and short runway.
• Runway as a first-class number. Track (max - current) / fill rate continuously during any downstream incident.
• Per-pipeline monitoring. Aggregate stats mask single-pipeline failure; alert per pipeline ID.
• Watch the oscillators. Pipelines that swing between near-empty and near-full during peaks are capacity candidates before they become incidents.
• Cold-start gating. Every queue alert should require JVM uptime over 300 seconds.

How Netdata helps

• Netdata collects the Logstash node stats API continuously, so events_count, queue_size_in_bytes, and event throughput appear as per-second trends rather than two hand-polled samples. Rate of change is visible directly on the curve.
• Because the agent also collects host CPU, JVM, disk, and network metrics on the same node, the three correlation patterns become a single-dashboard exercise: queue growth next to CPU, next to disk I/O on the PQ volume, next to output behavior.
• Baseline-relative anomaly detection fits this metric well, since healthy queue depth is workload-dependent and absolute thresholds are unreliable.
• Disk space and fill-rate trends on the queue volume turn the runway calculation into a graph instead of a manual estimate during an incident.
• Alerting on sustained occupancy tiers and growth, with uptime gating, maps directly onto the prevention rules above.

Related guides

• How Logstash actually works in production: a mental model for operators
• Logstash monitoring checklist: the signals every production pipeline needs
• Logstash monitoring maturity model: from survival to expert
• Logstash queue full: inputs blocked and the backpressure wedge