Logstash API unreachable on port 9600: crash, GC pause, or startup

Your liveness check against http://127.0.0.1:9600/ just started timing out or returning connection refused. That single fact tells you almost nothing by itself. The Logstash monitoring API shares the JVM with the pipeline, so an unreachable API means one of four things: the process is dead, the JVM is frozen in a garbage collection pause, Logstash is still starting up, or something is wrong with how the API is bound to the network.

The first two are very different incidents. A crash means zero event processing until something restarts the process. A GC pause means the process is alive, may recover on its own in seconds, and paging a human at 3 a.m. for it is usually wrong. This is why the playbook treats standalone API unreachability as a TICKET, not a PAGE: the benign short-lived causes are common enough that a fleet-wide page on this signal alone will train your team to ignore it.

There is also a trap in the opposite direction. A 200 OK from port 9600 only proves the JVM and its HTTP server are alive. It says nothing about whether the pipeline is processing events. The “living dead” scenario, where the API responds but output throughput has been zero for hours, is one of the most common Logstash monitoring failures. Liveness must always be paired with an output-rate check.

This guide covers how to tell the four causes apart quickly, what to check before restarting anything, and how to structure alerting so this signal is useful instead of noisy.

What this means

The monitoring API (Node Stats API) is an HTTP server inside the Logstash JVM, bound to port 9600 by default. Because it runs in the same process as the pipeline, it competes for the same heap, the same GC, and the same threads. Its reachability is therefore a JVM health signal, not a pipeline health signal.

The failure modes break down as follows:

flowchart TD
  A[API on 9600 unreachable] --> B{Process alive?}
  B -->|No| C[Crash: check logs, OOM killer, config errors]
  B -->|Yes| D{JVM uptime low?}
  D -->|Yes| E[Startup or restart in progress: wait and recheck]
  D -->|No| F{GC time high, heap near max?}
  F -->|Yes| G[GC pause or death spiral]
  F -->|No| H[Binding or config issue: port, host, SSL, API disabled]

Two details matter for how you respond. First, during a severe GC pause the API times out but the process recovers; if you restart in the middle of a long-but-survivable pause you convert a brown-out into a full outage plus queue replay. Second, in hardened deployments the API may be disabled entirely or bound to a non-default address or port, in which case “unreachable on 9600” is the expected state and you need process-level checks instead.

Common causes

Quick checks

Run these in order. All are read-only.

# 1. Is the process alive at all?
pgrep -f org.logstash.Logstash
systemctl is-active logstash

# 2. Distinguish refused vs timeout (this splits the cause list in half)
curl -sS --connect-timeout 5 http://127.0.0.1:9600/

# 3. Measure response time when it does connect
curl -s -o /dev/null -w "%{http_code} %{time_total}\n" http://127.0.0.1:9600/_node/stats

# 4. Who is actually listening on 9600?
ss -tlnp | grep 9600

# 5. How is the API configured?
grep -E 'api\.' /etc/logstash/logstash.yml

# 6. Recent crash evidence
journalctl -u logstash --since "15 min ago" | tail -n 50
dmesg | grep -i -E 'oom|killed process' | tail

Check 2 is the highest-value step. Connection refused means nothing is listening: crash, still starting, or bound elsewhere. Connection timeout with an alive process means the JVM is not scheduling the HTTP thread: GC pause or severe thread starvation.

Response time thresholds from operational experience: under 2 seconds is normal, sustained over 5 seconds indicates JVM stress, and no response for over 30 seconds on a running process is a severely impaired Logstash.

How to diagnose it

1. Confirm process state. If pgrep -f org.logstash.Logstash returns nothing, you are in the crash path. Go to step 4. If the process exists, continue.

2. Check JVM uptime. Once the API responds, read jvm.uptime_in_millis from /_node/stats/jvm. Uptime under ~2 minutes means you caught a startup or restart. Logstash startup legitimately takes 30-120 seconds, longer when replaying a persistent queue, and health checks that fire inside that window are false positives. Gate any alerting on uptime over 300 seconds.

3. Check GC and heap. Pull /_node/stats/jvm and look at mem.heap_used_percent and the gc.collectors.old and gc.collectors.young counts and times. Take two samples 60 seconds apart and compute GC overhead as delta(collection_time_in_millis) / delta(wall time). Over 10% is concerning, over 20% is severe, and over 50% with heap pinned near max is a GC death spiral: the process is alive but doing almost no useful work, and it will not recover on its own. If the API will not answer at all, jstack $(pgrep -f org.logstash.Logstash) still works against the live JVM (run it as the same user that owns the Logstash process, typically logstash), and GC logs, if enabled, give you pause history without the API.

4. If crashed, find out why before restarting. Tail /var/log/logstash/logstash-plain.log for the final error. Check the kernel log for OOM kills (journalctl -k if dmesg requires privileges). Common terminal causes are config errors at startup, plugin initialization failures, port conflicts, and persistent queue problems after an unclean shutdown. If the log shows checkpoint or page file errors after a crash, see Logstash won’t start after a crash: persistent queue corruption and checkpoint errors before deleting anything.

5. If the process is healthy but nothing listens on 9600, verify binding. Compare ss -tlnp output against the api.http.host and api.http.port settings in logstash.yml. The port is configurable, so do not assume 9600. The API can also be secured with SSL and authentication; a probe doing a plain HTTP GET against an SSL-enabled API will fail even though everything is fine. If the API is deliberately disabled, switch your liveness check to process and systemd state and rely on output throughput for functional health.

6. Capture evidence during the incident. If the API is intermittently reachable, grab /_node/hot_threads?threads=10&human=true a few times, seconds apart. Threads parked in GC-adjacent or blocked states corroborate heap pressure. The hot threads endpoint itself can be slow exactly when GC pressure is worst, which is itself diagnostic.

Metrics and signals to monitor

The critical design rule: page on sustained output-rate-zero, not on API unreachability. Temporary API stalls from GC pauses and restarts self-resolve. A sustained zero output rate with active input does not.

Fixes

Process crash

Find and fix the terminal cause in the log before restarting, or you will restart into the same failure. Config syntax errors, missing plugins, and permission problems all survive restarts. If the crash followed an unclean shutdown and the log implicates the persistent queue, follow the PQ corruption guide linked above; deleting queue or checkpoint files is destructive and can cause event loss or duplication.

GC pause or death spiral

A short pause needs no action; the process recovers. A confirmed death spiral (GC overhead over 50%, heap pinned, throughput near zero) is one of the few situations where an immediate restart is the correct move, because the JVM will not recover. With a persistent queue, in-flight events survive the restart. With a memory queue, whatever was queued is lost. After the restart, address the root cause: raise the heap in jvm.options (set -Xms and -Xmx equal; the old 1GB default is too small for most production workloads), reduce in-flight event volume via batch size and worker count, and enable -XX:+HeapDumpOnOutOfMemoryError so the next event leaves evidence. See the GC and heap sections of Logstash monitoring checklist: the signals every production pipeline needs for the thresholds to watch.

Slow startup

No fix to Logstash itself in most cases: startup of 30-120 seconds is normal, and persistent queue replay extends it. Fix the health checks instead. Give liveness probes a startup grace period of at least 120 seconds and gate alerts on jvm.uptime_in_millis > 300000. A too-aggressive probe during a rolling restart creates restart thrashing, which turns a slow start into a real outage.

Binding, SSL, or disabled API

Align your check with the actual configuration: correct port, correct scheme if SSL is enabled, credentials if basic auth is on. If the API is intentionally disabled, fall back to systemctl is-active logstash for liveness and to destination-side or agent-side throughput signals for functional health.

Prevention

• Pair liveness with throughput. Never alert on the API alone. The composite rule that catches real prolonged outages is output rate at zero while input rate is non-zero, sustained, with uptime over 300 seconds.
• Gate everything on uptime. Cold start produces low throughput, high CPU, and an unresponsive API for the first minute or two. Every liveness and throughput alert should include an uptime gate.
• Track the post-GC heap floor, not the peak. Alerting on heap_used_percent > 80% fires on every normal GC peak and gets silenced; the rising post-GC floor is the early warning that the API will start timing out next week, not tonight.
• Monitor restart count, not just current state. In containerized deployments the orchestrator restarts Logstash fast enough to hide crash loops from a process-exists check.
• Do not poll the API faster than every 10 seconds. On extremely loaded instances the metrics API itself adds load, and aggressive polling makes the symptom you are watching worse.
• Know your deployment’s API posture. If your hardening disables the API or moves it off 9600, document that in the runbook so the on-call does not chase a phantom “API down” incident.

How Netdata helps

Netdata shortens the triage above by putting the correlating signals on one timeline instead of four different tools:

• API availability and JVM metrics together. Netdata’s Logstash collector polls the node stats endpoints, so API reachability gaps line up visually with heap, GC, and uptime curves. You can see in one view whether the gap was a restart (uptime reset), a GC event (old-gen time spike), or a hard crash (data stops entirely).
• GC overhead as a rate. Young and old collector times are graphed per second, making the 10% and 20% overhead thresholds directly readable instead of requiring manual delta math.
• Uptime and restart detection. Uptime resets are obvious on the chart, which catches container-masked crash loops that a process check misses.
• Output throughput next to liveness. Because pipeline flow metrics are collected alongside JVM stats, you can immediately answer the question that decides severity: is the pipeline still delivering, or is this a living-dead process?
• Uptime-gated, baseline-relative alerts. Alerting on deviation from rolling baselines with uptime conditions avoids the false positives that make standalone API checks page-worthy for the wrong reasons.

Related guides

• How Logstash actually works in production: a mental model for operators
• Logstash monitoring checklist: the signals every production pipeline needs
• Logstash monitoring maturity model: from survival to expert
• Logstash won’t start after a crash: persistent queue corruption and checkpoint errors
• Logstash memory queue vs persistent queue: durability, visibility, and failure modes
• Logstash config reload failed: reloads.failures and invisible configuration drift