Logstash output errors and retries: reading downstream failure before the queue grows

Output-side failures are the most common cause of Logstash queue growth and eventual outage. When the downstream destination (Elasticsearch, Kafka, an HTTP endpoint) starts rejecting, timing out, or slowing down, Logstash output plugins retry. Those retries preserve data temporarily but hide mounting delay. The pipeline appears alive while events accumulate.

The critical window for diagnosis is between the first output errors and the moment the queue fills. Once the queue reaches capacity, inputs are blocked, upstream systems buffer or drop events, and the incident has cascaded beyond Logstash.

What this means

Output plugins handle downstream failures by retrying. The Elasticsearch output retries all non-200 responses from the bulk API indefinitely, with exponential backoff starting at 2 seconds (retry_initial_interval) and capped at 64 seconds (retry_max_interval). The HTTP output retries indefinitely for codes in its retryable_codes list (default: 429, 500, 502, 503, 504) when retry_failed is enabled, which is the default.

Brief retries during downstream failover (a rolling restart, a network blip) self-heal and are expected. The problem starts when retries become sustained. Each retry occupies a worker thread. While workers are stuck retrying, they cannot pull new batches from the queue. The queue stops draining. If input continues, the queue grows.

flowchart TD
    A["Output destination slow or rejecting"] --> B["Plugin receives 429, timeout, or connection error"]
    B --> C["Plugin retries with exponential backoff"]
    C --> D{"Retry succeeds within seconds?"}
    D -- "Yes" --> E["Brief failover - self-heals"]
    D -- "No, sustained" --> F["Worker threads blocked on output"]
    F --> G["Queue stops draining"]
    G --> H["Queue grows"]
    H --> I["Inputs blocked by backpressure"]
    H --> J["PQ fills toward max_bytes"]
    J --> I

Two patterns emerge:

Retries absorbing impact temporarily. Output errors are visible but the queue is not growing. The retry mechanism is masking the downstream problem. Act here.

Retries no longer absorbing. Queue growth turns positive and stays positive. Workers are blocked, the queue is filling, and the clock is ticking toward input backpressure. Severity escalates when output rate decline and queue growth coincide.

The worst-case variant is the persistent queue masking a real outage. PQ can absorb hours of output failure before hitting max_bytes. During that window, process health checks pass, pipeline status shows “running,” and throughput metrics may still look reasonable. The only advance warnings are rising PQ occupancy and the output error logs.

Common causes

Quick checks

# Check cumulative input vs output event counts (compare two snapshots for rate)
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -E '"in"|"out"|"filtered"'

# Grep the log for output-side errors
grep -Ei '(retry|error|exception|failed|reject|unavailable|timeout|429|503)' /var/log/logstash/logstash-plain.log | tail -n 200

# Check per-output plugin event stats (look for bulk_requests.failures, documents.non_retryable_failures)
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | python3 -c "
import sys,json
pipes = json.load(sys.stdin)['pipelines']
for name, pipe in pipes.items():
    for o in pipe.get('plugins',{}).get('outputs',[]):
        print(f'Pipeline={name} Output={o.get(\"name\",\"?\")} id={o.get(\"id\",\"?\")}')
        print(json.dumps(o.get('events',{}), indent=2))
        if 'bulk_requests' in o:
            print(json.dumps(o['bulk_requests'], indent=2))
"

# Check queue occupancy and growth
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | python3 -c "
import sys,json
pipes = json.load(sys.stdin)['pipelines']
for name, p in pipes.items():
    q = p.get('queue',{})
    print(f'Pipeline={name} type={q.get(\"type\",\"?\")} events={q.get(\"events_count\",0)} bytes={q.get(\"queue_size_in_bytes\",0)} max={q.get(\"max_queue_size_in_bytes\",\"N/A\")}')
"

# Check DLQ size (if enabled)
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -A5 dead_letter_queue

# Check output destination health (Elasticsearch example)
curl -sS --max-time 5 localhost:9200/_cluster/health?pretty

# Check hot threads for output blocking patterns
curl -sS 'http://127.0.0.1:9600/_node/hot_threads?threads=10&human=true'

How to diagnose it

1. Confirm the output rate has dropped. Compare flow.output_throughput.current against input throughput. If output is below input and the gap is sustained, the pipeline is falling behind. Cross-reference with JVM uptime to rule out cold-start effects (gate on uptime greater than 300 seconds).

2. Identify the error type from logs. The specific error tells you the cause:

   • HTTP 429: destination is rate limiting or thread_pool saturated
   • Connection timeout or refused: network issue or destination down
   • SSL/TLS/certificate: auth failure or expired cert
   • HTTP 400/404 from Elasticsearch: mapping conflict or missing index (these go to DLQ if enabled, or are silently lost if not)

3. Check per-output plugin stats for document-level failures. The Elasticsearch output exposes bulk_requests.failures, bulk_requests.with_errors (partial failures within a successful bulk request), and documents.non_retryable_failures. A with_errors count alongside HTTP 200 responses means silent partial data loss: the bulk request succeeded but individual documents were rejected.

4. Assess queue runway. If using persistent queues, calculate time-to-full from the fill rate:

   remaining = max_queue_size_in_bytes - queue_size_in_bytes
   growth_rate = flow.queue_persisted_growth_bytes.current
   runway_seconds = remaining / growth_rate  (if growth_rate > 0)
   

   If runway is under 30 minutes with sustained growth, inputs will be blocked soon.

5. Check whether DLQ is absorbing failures. If DLQ is enabled and growing, specific events are being permanently rejected. If DLQ is disabled (the default), permanently failed events are silently lost after retries are exhausted. Both are data integrity problems.

6. Confirm the downstream is actually impaired. Query the destination independently. Do not rely solely on Logstash metrics. The cascade (Elasticsearch rejects, Logstash retries, queue fills, inputs block, Beats buffer, upstream monitoring goes dark) is only diagnosable if both sides are checked.

7. Use hot threads to confirm output blocking. Workers stuck in BLOCKED or TIMED_WAITING state within output plugin code confirms the output is the bottleneck, not a filter. CPU will be low because workers are waiting on I/O, not computing.

Metrics and signals to monitor

Fixes

Fix the downstream issue first

Output retries are a symptom. Check:

• Elasticsearch cluster health (red or yellow status causes slow indexing and rejections)
• Elasticsearch write thread_pool queue depth and rejections
• Network connectivity and latency to the destination
• Destination disk space and shard allocation

If the destination is temporarily down for planned maintenance, Logstash with PQ will buffer. Calculate runway and confirm the destination will recover before the queue fills.

Enable DLQ to prevent silent data loss

DLQ is disabled by default. Without it, events that permanently fail output delivery after all retries are logged and silently lost. Enable it in logstash.yml:

dead_letter_queue.enable: true

For the Elasticsearch output, only HTTP 400 and 404 responses are sent to DLQ by default (non-retryable errors). All other non-200 responses are retried indefinitely. You can add custom codes via dlq_custom_codes:

output {
  elasticsearch {
    # ... other settings ...
    dlq_custom_codes => [413]
  }
}

Monitor DLQ size. A DLQ that hits max_queue_size_in_bytes (default 1GB) starts dropping events permanently based on the configured storage policy (drop_newer or drop_older).

Limit retries for known unrecoverable patterns

The Elasticsearch output has no max_retries parameter. It was removed in v8.0.0 of the plugin.

All non-200 responses except 400 and 404 are retried indefinitely. For known unrecoverable error types, use drop_error_types (available in ES output v12.1.0+) to stop retrying after specific error types instead of looping forever.

For the HTTP output, setting retry_failed => false disables plugin-level indefinite retry on response codes. automatic_retries (default 1) still applies for connection-level errors only. This prevents infinite retry loops at the cost of data loss on persistent failures.

Shed load or isolate non-critical pipelines

If the queue is filling faster than the downstream can recover:

1. Stop or reduce input on non-critical pipelines to preserve runway for critical data
2. In multi-pipeline deployments, isolate the affected pipeline’s queue to prevent cross-pipeline JVM heap contention
3. If using Kafka input, pause consumer groups for non-critical topics to reduce ingest pressure

These are stopgap measures. The downstream issue must still be fixed before the queue can resume draining.

Prevention

• Alert on output error rate proactively. Do not wait for queue growth. Sustained non-zero retry or error patterns in the logs warrant investigation.
• Correlate output rate with queue growth. Declining output throughput combined with positive queue growth is the definitive downstream backpressure signal. Alert on both conditions together.
• Calculate PQ runway continuously. (max_queue_size - queue_size) / growth_rate tells you time-to-full. Alert when runway drops below 30 minutes with sustained growth.
• Monitor per-output document-level stats. documents.non_retryable_failures and bulk_requests.with_errors catch silent partial data loss that throughput metrics miss.
• Treat DLQ growth as a data integrity incident. DLQ events require manual replay through a separate pipeline using the dead_letter_queue input plugin.
• Monitor the destination independently. Elasticsearch cluster health, bulk rejection rate, and indexing latency should be monitored alongside Logstash metrics. The failure cascade crosses system boundaries.
• Use baseline-relative thresholds for throughput. Absolute events-per-second thresholds become meaningless as workloads change. Alert on deviation from rolling averages instead.

How Netdata helps

• Per-second output throughput and queue growth metrics show divergence from input rate before the queue compounds.
• ML-based anomaly detection on flow.output_throughput, flow.queue_backpressure, and queue.queue_size_in_bytes surfaces sustained retry patterns that fixed thresholds miss.
• Correlated timelines of output plugin duration, queue growth, and log error patterns distinguish a downstream outage (output duration up, CPU low, queue growing) from a compute bottleneck (CPU high, no output errors, queue growing).
• DLQ size monitoring flags the silent data loss path that often goes unnoticed.
• Per-pipeline and per-plugin breakdown prevents aggregate metrics from masking one failing output among several healthy ones.

Related guides

• Logstash address already in use: input port conflicts on Beats, TCP, and HTTP
• Logstash API unreachable on port 9600: crash, GC pause, or startup
• Logstash Beats input: Filebeat backpressure and connection health
• Logstash certificate expiry: the silent, total outage no built-in metric shows
• Logstash configuration drift: when the running config no longer matches the deployed one
• Logstash configuration integrity: detecting unexpected changes to pipeline files
• Logstash config reload failed: reloads.failures and invisible configuration drift
• Logstash CPU-bound filters (grok hell): high CPU, saturated workers, growing queue
• Logstash could not be started: another instance is using the configured data.dir
• Logstash disk full: PQ, DLQ, and log volumes competing for space
• Logstash file descriptor pressure: leaks, tailed files, and reconnection churn
• Logstash file input and sincedb: re-read loops, duplicates, and FD pressure