Logstash Elasticsearch 429: retrying failed action with response code 429

The log line retrying failed action with response code: 429 from logstash.outputs.elasticsearch means Elasticsearch refused a bulk request because its write thread pool queue was full (es_rejected_execution_exception). ES is saturated and cannot keep up with the incoming bulk request rate.

This is the most common trigger of the backpressure wedge in Logstash-to-ES pipelines. The ES output plugin retries the rejected batch with exponential backoff. Worker threads block on those retries. The internal queue fills. Inputs are backpressured, and upstream systems like Filebeat buffer locally or begin dropping events.

This is not a Logstash configuration problem. Logstash is doing what it should. The fix is downstream in Elasticsearch indexing capacity. Retry settings can reduce log noise or slow the cascade, but the root cause is ES throughput.

Events are not lost during 429 rejections. The ES output retries indefinitely. But delay mounts steadily, and if your persistent queue fills to capacity, inputs block.

What this means

flowchart TD
    A["ES indexing capacity saturated"] --> B["Bulk thread pool queue full"]
    B --> C["ES returns HTTP 429"]
    C --> D["LS retries with backoff: 2s to 64s"]
    D --> E["Worker threads block"]
    E --> F["Internal queue grows"]
    F --> G["Inputs backpressure"]
    G --> H["Upstream buffers or drops"]

The ES output plugin sends events to Elasticsearch in bulk requests. When Elasticsearch’s write thread pool queue fills (configurable via thread_pool.write.queue_size in elasticsearch.yml; the bulk pool was renamed write in ES 6.3 and the thread_pool.bulk.* fallback was removed in 7.0; the default queue size was 1000 in early 7.x and 10000 in later 7.x and 8.x), the request is rejected with HTTP 429.

The 429 response body carries an exception type that distinguishes two root causes:

• es_rejected_execution_exception: the write thread pool queue is full. ES cannot enqueue the request. Most common trigger.
• circuit_breaking_exception: the request would exceed ES’s JVM memory circuit breaker. The bulk request is too large relative to available heap.

The Logstash ES output treats both as retryable. There is no retry_on_429 setting and no max_retries limit that applies to 429 specifically. The plugin retries indefinitely with exponential backoff: retry_initial_interval defaults to 2 seconds, doubling on each retry up to retry_max_interval of 64 seconds. Under sustained 429 pressure, each worker backs off to at most one bulk request every 64 seconds, and effective throughput collapses.

The retry log message is emitted at INFO level. Sustained repetition indicates a real downstream problem, but individual messages are normal backpressure signaling, not a crash or data loss event.

Common causes

Quick checks

Log paths vary by installation. Adjust /var/log/logstash/logstash-plain.log to match your deployment.

# Count 429 rejections in Logstash logs
grep -c "response code: 429" /var/log/logstash/logstash-plain.log

# Check pipeline event flow, queue depth, and output stats
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty

# Check ES cluster health
curl -sS http://127.0.0.1:9200/_cluster/health?pretty

# Check ES write thread pool (bulk): look at rejected column
curl -sS "http://127.0.0.1:9200/_cat/thread_pool/write?v&h=node_name,active,queue,rejected,completed"

# Check ES indexing stats for failure counts
curl -sS http://127.0.0.1:9200/_stats/indexing?pretty

# Check what Logstash worker threads are actually doing
curl -sS 'http://127.0.0.1:9600/_node/hot_threads?pretty'

# Check ES output plugin bulk request stats
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | grep -A20 '"outputs"'

How to diagnose it

1. Confirm the 429 source type. Grep the log for the exception type in the response body. es_rejected_execution_exception means thread pool saturation. circuit_breaking_exception means ES heap pressure from oversized or too many concurrent requests.

# Distinguish rejection types (requires GNU grep for -P flag)
grep "response code: 429" /var/log/logstash/logstash-plain.log | grep -oP '"type"=>"[^"]*"' | sort | uniq -c

2. Check ES cluster health and write thread pool. If _cluster/health returns yellow or red, indexing is impaired by shard allocation, not just load. Check _cat/thread_pool/write for the rejected column. Any non-zero and growing rejected count confirms ES-side saturation.

3. Check Logstash queue and throughput. Pull _node/stats/pipelines and compare flow.input_throughput against flow.output_throughput. If output is consistently below input, the queue is growing. Check queue.events_count and, for persistent queues, queue.queue_size_in_bytes relative to queue.max_queue_size_in_bytes.

4. Estimate PQ runway. If you use persistent queues, calculate how long until the queue fills: (max_queue_size_in_bytes - queue_size_in_bytes) / growth_rate_in_bytes_per_second. This tells you whether you have minutes or hours before inputs are blocked.

5. Confirm the output-blocking pattern with hot threads. Run _node/hot_threads. If workers are in BLOCKED or TIMED_WAITING state inside output plugin code (not filter code), this confirms the downstream bottleneck. CPU should be low because workers are I/O-waiting, not computing.

6. Check ES-side indexing capacity. Pull _stats/indexing and compare the indexing rate against your Logstash output rate. If ES indexing throughput is the ceiling, the problem is shard count, hardware, or cluster size, not Logstash.

Metrics and signals to monitor

Fixes

Fix Elasticsearch indexing capacity (the actual fix)

This is where the problem lives and where the fix belongs.

• Scale the ES cluster. Add data nodes to distribute indexing load. More shards for hot indices increases parallelism.
• Check for mapping explosion. If the target index has tens of thousands of fields, every bulk request is expensive. Audit field count and mapping size. Dynamic mapping with uncontrolled field creation is a common cause.
• Check ES heap and GC. If ES nodes are in their own GC pressure, indexing throughput drops. Monitor ES JVM stats independently.
• Check disk pressure on ES data nodes. ES enforces the flood-stage watermark at 95% disk, setting index.blocks.read_only_allow_delete on indices. At the low watermark (85% by default), ES stops allocating new shards to the node. At the high watermark (90%), ES begins relocating shards away, which adds I/O load and can slow indexing.

Increase ES bulk queue size (temporary relief, use with caution)

Increasing thread_pool.write.queue_size in elasticsearch.yml lets ES queue more bulk requests before rejecting them. This is a static setting and requires an ES node restart.

This does not increase indexing throughput. It only delays rejections. Worse, queued requests consume ES heap. If you raise the queue size without addressing indexing capacity, you risk trading 429 rejections for circuit breaker trips or ES OOM. Use this only as a stopgap while scaling ES.

Reduce Logstash output concurrency (buy time)

If you cannot fix ES immediately and PQ runway is short, reduce the load Logstash places on ES. Both changes below require a pipeline restart or reload:

• Reduce pipeline.workers for the affected pipeline. Fewer workers means fewer concurrent bulk requests. This directly reduces ES thread pool pressure but also reduces Logstash throughput.
• Reduce pipeline.batch.size. Smaller batches mean smaller individual bulk requests, which are less likely to trip circuit breakers but more numerous.

These are stopgaps. They slow the cascade but do not fix the root cause.

Suppress 429 log noise

If the 429 messages are filling your logs and you are confident the retries are expected behavior during a known ES capacity event, suppress them:

output {
  elasticsearch {
    # ... your existing config ...
    silence_errors_in_log => ["es_rejected_execution_exception"]
  }
}

The setting takes Elasticsearch error types, not HTTP status codes, so for 429 rejections you must list es_rejected_execution_exception (add circuit_breaking_exception if the circuit breaker is the trigger). If you are on an older plugin version (before v11.7.0), the setting is named failure_type_logging_whitelist instead. It still works but prints a deprecation warning. Migrate to silence_errors_in_log when you can.

Suppressing the log does not change retry behavior. The output still retries indefinitely. Use this only when you are actively managing the ES-side issue and the log volume is itself causing problems.

Calculate PQ runway and shed load if needed

# Estimate PQ occupancy (bytes)
# Replace pipeline ID as needed
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | \
  python3 -c "
import sys, json
d = json.load(sys.stdin)['pipelines']
for pid, p in d.items():
    q = p.get('queue', {})
    used = q.get('queue_size_in_bytes', 0)
    mx = q.get('max_queue_size_in_bytes', 0)
    if mx > 0:
        print(f'{pid}: {used/1024/1024:.0f}MB / {mx/1024/1024:.0f}MB ({used/mx*100:.1f}%)')
"

If runway is short (under 30 minutes at current growth rate), shed non-critical pipelines or reduce incoming load before the queue fills. Once PQ hits max_bytes, inputs are blocked immediately and upstream systems begin failing.

Prevention

• Monitor ES bulk rejection rate alongside Logstash output rate. The rejected count in _cat/thread_pool/write is the leading indicator. Catch it before the Logstash queue starts growing.
• Track PQ fill rate, not just occupancy. A PQ at 60% that is growing steadily is more urgent than one at 85% that is stable. Compute runway from growth rate, not from absolute occupancy.
• Set up composite alerting. Alert when output rate is below input rate AND queue is growing AND output errors or retries are non-zero. Individual signals in isolation are noisy. The composite catches the cascade early.
• Size ES with headroom for peak indexing. If ES indexing throughput at peak is within 10% of capacity, you are one traffic spike away from 429s.
• Monitor ES cluster health proactively. Yellow or red cluster status causes slow indexing, which causes 429s. The ES problem starts before the Logstash symptom appears.
• Watch for mapping explosion. Audit field counts in your indices regularly. Dynamic mapping without controls can cause indexing throughput to degrade gradually over weeks.

How Netdata helps

• Per-second pipeline throughput catches the input/output divergence before the queue grows visibly. Per-second resolution means you see the gap forming within seconds, not minutes.
• Queue occupancy trend and growth rate provide runway estimation for PQ-based deployments. Correlating fill rate with output rate decline tells you whether the queue is growing because of downstream pressure or compute saturation.
• Worker utilization and CPU correlation distinguishes output blocking (low CPU, high utilization, workers on I/O wait) from filter saturation (high CPU, high utilization). If CPU is low and the queue is growing, the problem is downstream.
• ES-side metrics (bulk rejections, cluster health, indexing rate) correlate with Logstash output rate on a single timeline when both are monitored. The causal chain becomes visible without switching tools.
• Composite alerting on sustained output rate decline with non-zero input rate catches the backpressure cascade at its earliest stage, before queue growth becomes critical.
• Hot threads collection from the Logstash API during incidents provides forensic evidence of worker thread state without manual intervention.

Related guides

• Logstash Beats input: Filebeat backpressure and connection health
• Logstash CPU-bound filters (grok hell): high CPU, saturated workers, growing queue
• Logstash disk full: PQ, DLQ, and log volumes competing for space
• Logstash API unreachable on port 9600: crash, GC pause, or startup
• Logstash configuration drift: when the running config no longer matches the deployed one