Logstash could not be started: another instance is using the configured data.dir

Logstash exits immediately at startup with an error like:

Logstash could not be started because there is already another instance
using the configured data.dir. Please change the value of path.data or
configure a different instance.

The wording varies slightly by version, but the meaning is always the same: Logstash tried to take an exclusive lock on its data directory and failed. This is a hard startup block. No pipeline loads, no events flow, the process exits.

There is a small, enumerable set of causes: a real second process holds the lock, a stale lock survived an unclean shutdown, or two instances were pointed at the same directory by mistake. Figure out which one you have before deleting anything, because deleting the lock while a live instance holds it is exactly the corruption scenario the lock exists to prevent.

What this means

Since Logstash 6.0, the JVM takes an exclusive file lock on a file named .lock inside the directory configured by path.data at startup, using Java NIO FileChannel.tryLock(), which maps to an OS-level advisory file lock. If another process already holds it, tryLock() returns null and Logstash aborts. The lock exists because two instances writing to the same persistent queue, checkpoints, or sincedb files will corrupt each other’s state. Before the lock existed, running two instances against one data directory was a known PQ-corruption path.

Where path.data lives depends on the install:

• RPM/DEB packages: /var/lib/logstash
• Official Docker image: /usr/share/logstash/data
• Tarball/zip: data under the Logstash home directory
• Anything overridden by path.data in logstash.yml or --path.data on the command line

The file is hidden (leading dot), so a casual ls of the data directory will not show it. This trips people up: they look in the directory, see nothing wrong, and conclude the error is spurious.

Common causes

Quick checks

All read-only and safe.

# 1. Is any Logstash JVM actually running on this host?
pgrep -af org.logstash.Logstash
# or
ps aux | grep '[o]rg.logstash.Logstash'

# 2. What does the service manager think?
systemctl status logstash

# 3. Does the lock file exist, and how old is it?
ls -la /var/lib/logstash/.lock        # adjust to your path.data
stat /var/lib/logstash/.lock

# 4. Which process (if any) has the lock file open?
lsof /var/lib/logstash/.lock
fuser -v /var/lib/logstash/.lock

# 5. In containers: is another container mounting the same volume?
docker ps --format '{{.Names}} {{.Mounts}}' | grep logstash

# 6. Is path.data on a network or shared filesystem?
findmnt -T /var/lib/logstash

Interpretation notes:

• If step 1 shows a live process, stop here. The lock is doing its job. Do not delete the file.
• lsof/fuser on the .lock file is the most direct evidence of a local holder. An empty result plus an existing file plus no Logstash process points at a stale lock.
• stat gives you mtime. A .lock whose timestamps predate the last crash is consistent with a leftover.
• The kernel releases OS-level file locks when the holding process dies, even on kill -9. A truly stale lock from a dead local process is rarer than people assume; more often the “stale” case is a holder in another mount namespace or container, a holder on another host via NFS, or version-specific lock handling. Treat “the process is definitely gone” as something to prove, not assume.

How to diagnose it

Work through these in order. The goal is to distinguish “live holder” from “leftover” before touching anything.

1. Confirm no local Logstash process exists. Use pgrep -af org.logstash.Logstash, not just systemctl status. A manually started instance, a second unit (logstash@foo.service), or a containerized instance on the same host will not show in the default unit’s status.

2. Check containers and other namespaces. A container with a host-path volume mounted at path.data holds the lock from inside its own namespace. pgrep on the host still sees the Java process, but operators miss it because they are thinking in terms of the systemd unit. Check docker ps and pod status on the node.

3. Check for a remote holder on shared storage. If findmnt shows NFS or another network filesystem under path.data, the holder may be a Logstash on a different host. No local command will show it. Check your inventory: is any other node configured with the same data path?

4. Correlate with the last shutdown. Pull the end of the previous run: journalctl -u logstash --since "-2 hours" and, for OOM kills, journalctl -k or dmesg. If the last shutdown was an OOM kill, SIGKILL, or node failure, the stale-lock explanation gains weight. If it was a clean SIGTERM, look harder for a live holder.

5. Only now decide: remove the lock, or fix the configuration. If you found a live holder or a duplicate config, the lock file is innocent and deleting it changes nothing.

flowchart TD
  A[Startup fails: data.dir lock error] --> B{Logstash process running anywhere?}
  B -->|Yes, local| C[Duplicate instance or service - stop it, do not delete .lock]
  B -->|Yes, in container| D[Container holds lock via mounted volume - stop container or fix mounts]
  B -->|No local process| E{path.data on shared/network FS?}
  E -->|Yes| F[Check other hosts mounting the same path - remote holder]
  E -->|No| G{Last shutdown unclean? crash, kill -9, OOM}
  G -->|Yes| H[Verify with lsof/fuser that nothing holds .lock, then remove stale .lock]
  G -->|No| I[Recheck: hidden second unit, cron job, orchestrator restart loop]

Metrics and signals to monitor

Once Logstash is back up, a few signals tell you whether the underlying cause (usually an unclean shutdown) will recur:

Fixes

Stale lock, no live process

Once you have positively confirmed nothing holds the lock (steps 1-4 above):

# DESTRUCTIVE if a live instance exists. Verify first with lsof/fuser.
rm /var/lib/logstash/.lock
systemctl start logstash

Then watch startup: journalctl -u logstash -f. A clean start loads pipelines within a minute or two on most hosts. If it fails again with checkpoint or page errors, the unclean shutdown also damaged the persistent queue. That is a separate recovery path; see the PQ corruption guide linked below.

A real duplicate instance

Find why two things are trying to run Logstash:

• A hand-started debugging process that was never stopped: kill it cleanly (SIGTERM) and let systemd own the service.
• A duplicate unit or an old init script alongside systemd: disable and remove one.
• Two containers scheduled onto the same node with the same hostPath volume: fix the volume layout so each instance gets its own data directory. Per-instance path.data is the supported way to run multiple instances on one host.

Two intentional instances, one data dir

Give each instance a distinct path.data, either in its own logstash.yml or via --path.data. Note that running multiple pipelines does not require this: pipelines defined in pipelines.yml share the single instance’s data directory and one lock, which is fine. The one-lock-per-data-dir rule applies to JVM processes, not pipelines.

Containers with persistent volumes

If a container was force-killed, the .lock persists in the mounted volume because the volume survives the container. Delete the file on the host (same verification rules apply) or handle cleanup in an entrypoint wrapper. The cleaner fix is a real termination grace period so Logstash shuts down on SIGTERM instead of being SIGKILLed at the deadline, which avoids both the stale lock and PQ corruption in one move.

NFS or shared storage under path.data

Java file locking over NFS is historically unreliable: locks can fail spuriously or appear to succeed when another host holds them. If you must keep path.data on NFS, verify holders across all mounting hosts before removing anything. The operationally saner answer is local disk for path.data. NFS under the persistent queue also amplifies drain and recovery times, so this aligns with PQ guidance generally.

Prevention

• Stop Logstash cleanly. SIGTERM and an adequate stop timeout (TimeoutStopSec in the unit, termination grace period in Kubernetes) let Logstash release the lock and checkpoint the PQ. Reserve SIGKILL for a genuinely hung process.
• Fix the OOM that caused the crash. If the unclean shutdown was an OOM kill, size the heap honestly: -Xms equal to -Xmx, and keep heap to roughly half of system memory so off-heap and page cache have room. Monitor the post-GC floor, not raw heap percent.
• One data dir per instance, everywhere. Make it a config-management invariant: no two units, containers, or hosts ever share path.data. For containers, that means distinct volumes per replica, never a shared hostPath.
• Alert on unexpected restarts. JVM uptime resets and container restart counts are cheap signals that surface the crash before the lock error. Catching the first unclean exit beats debugging the lock at the second one.
• Keep path.data on local disk unless you have a specific reason and have tested failover behavior.

How Netdata helps

Netdata will not unlock the file, but it shortens the path to understanding why the error keeps coming back:

• Process and uptime tracking for the Logstash JVM makes unexpected restarts and crash loops visible immediately, the leading indicator for stale-lock incidents.
• JVM heap and GC correlation (heap used, per-pool old-gen pressure, GC time versus wall clock) shows whether the previous death was memory-driven, so you fix the OOM instead of deleting the lock each time.
• Container restart counts and cgroup memory on the node surface force-killed containers and OOM events that systemd-level checks miss.
• Node Stats API collection (pipelines, JVM, process stats from port 9600) gives post-recovery confirmation that pipelines actually came back and are draining the persisted queue, not just that the process exists.
• Alerting on the startup error string turns “Logstash has been down since 3 a.m.” into a page at the first failed start, which matters because this error is a total outage with zero event flow.

Related guides

• Logstash won’t start after a crash: persistent queue corruption and checkpoint errors
• Logstash memory queue vs persistent queue: durability, visibility, and failure modes
• Logstash monitoring checklist: the signals every production pipeline needs
• How Logstash actually works in production: a mental model for operators
• Logstash config reload failed: reloads.failures and invisible configuration drift