ZooKeeper KeeperErrorCode = NodeExists: create failing on an already-created znode

The error:

org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /some/path

A create() hit a znode path that already exists. ZooKeeper classifies this as a state exception, not a system fault. The cluster refused to overwrite an existing node, exactly as specified. The question is whether the caller expected that path to be free.

Most production hits follow one of two patterns. The first is a retry after ConnectionLoss: the original create() committed but the response was lost in flight. The client does not know whether the operation succeeded, retries, and receives NodeExists. The second is a genuine race: two candidates trying to create the same ephemeral leader or lock node, where exactly one wins and the other should lose gracefully. Both are expected. The incident starts when a client mishandles the exception, or when an orphaned ephemeral blocks the legitimate owner indefinitely.

This is almost never a server-side fault. The fix is usually in how the caller treats ConnectionLoss as an unknown outcome rather than a confirmed failure.

What this means

ZooKeeper distinguishes system errors (ConnectionLoss, SessionExpired, OperationTimeout) from state exceptions (NodeExists, NoNode, BadVersion). State exceptions describe real cluster state: the operation was processed against an authoritative view, and the answer is “the world is not what you assumed.” System errors mean the client cannot determine the outcome.

ConnectionLoss is the one code where the outcome is genuinely unknown. The request may have been dropped before reaching the leader, reached the leader and been proposed but not committed, or committed with the response lost on the return path. The ZooKeeper client library does not retry non-idempotent operations automatically for this reason. A create() on an existing path is not the same operation when retried: the second call returns NodeExists even if the first one succeeded.

flowchart TD
    A[Client create /path] --> B{Connection breaks?}
    B -->|no| C[Success]
    B -->|yes - outcome unknown| D[Client sees ConnectionLoss]
    D --> E[Client retries create]
    E --> F{Path exists?}
    F -->|no - first call dropped| C
    F -->|yes - first call committed| G[NodeExists returned]
    G --> H{Client logic}
    H -->|check ephemeralOwner vs session| I[Recovers cleanly]
    H -->|retry blindly| J[Sustained retry storm]

For ephemeral nodes, every znode carries the ID of the session that created it (the ephemeralOwner field in its stat structure). When a retry returns NodeExists, the caller can read the owner and compare it against its own session ID. A match means “this is my node, the original create succeeded.” A mismatch means “a different session holds this path”: either a peer that won a race, or a previous incarnation of this client whose session has not yet been reaped.

Common causes

CauseWhat it looks likeFirst thing to check
ConnectionLoss retry raceSingle NodeExists after a connection drop or leader election; client logs show a preceding ConnectionLossget /path in zkCli, compare ephemeralOwner to current session ID
Concurrent candidate raceTwo or more clients starting simultaneously; one wins, others see NodeExists once and back offApplication logs for leader-election or lock-acquisition code paths
Orphaned ephemeral from prior sessionReconnect keeps hitting NodeExists; owner session is neither current nor remembered; node persists past expected timeoutCompare ephemeralOwner to recently expired sessions; check dump output
Persistent node from a previous deployNode is persistent, not ephemeral; survives all session expirations; NodeExists never self-resolvesget /path in zkCli: ephemeralOwner is 0 for persistent nodes
Bug in retry loopNodeExists rate sustained above zero; same client producing it repeatedly; no peer racingApplication retry code: is ConnectionLoss treated as “retry blindly” rather than “unknown outcome”?

Quick checks

All commands below are read-only. The four-letter words (mntr, dump, isro) require whitelisting via 4lw.commands.whitelist on ZooKeeper 3.5.3+.

# Check the offending znode's stat, including ephemeralOwner.
# Connect from a host with zkCli.sh installed.
zkCli.sh -server localhost:2181
# At the prompt:
#   get /some/path
# The output shows: cZxid, mtime, version, ephemeralOwner, ...
# ephemeralOwner = 0 means persistent node; non-zero is the owning session ID.

# Is the cluster healthy enough that the retry storm is a symptom, not the cause?
echo mntr | nc localhost 2181 | grep -E 'zk_server_state|zk_outstanding_requests|zk_num_alive_connections|zk_ephemerals_count'

# Is the cluster mid-election? Elections force clients through reconnect.
echo mntr | nc localhost 2181 | grep -E 'zk_server_state'

# List ephemeral nodes and their owning sessions (expensive on large trees).
echo dump | nc localhost 2181 | grep -i '/some/path'

# Confirm functional state. A node in "ro" mode cannot commit writes.
echo isro | nc localhost 2181

How to diagnose it

  1. Identify the failing path and node type. Pull the exact /some/path from the client stack trace. Connect with zkCli.sh and run get /some/path. If ephemeralOwner is 0, the node is persistent: someone created it in a previous deploy and forgot to clean it up. If non-zero, it is ephemeral and the value is the owning session ID.

  2. Check whether the owner is the current client. Get the current client’s session ID. Most client libraries expose it; zkCli.sh prints it in the connect banner. Compare it to ephemeralOwner. A match means the original create() succeeded and the retry was unnecessary. The client should treat this as success, not failure.

  3. Check whether the owner session is still alive. If the owner is not the current client, look it up in dump output, which lists sessions and their ephemerals. If the owning session is gone from the session table but the ephemeral still exists, the cleanup raced with expiry. The leader’s session tracker will reap it on the next expiry sweep. If it persists beyond the full session-timeout window, that is a bug worth filing upstream.

  4. Determine whether ConnectionLoss preceded the NodeExists. Check the client logs for ConnectionLoss or SessionExpired immediately before the NodeExists. If present, the cause is the retry race, not a real conflict. Cross-reference with the server’s mntr output around the same timestamp to confirm connection instability or leader election.

  5. For Kafka broker registration specifically. If the path is /brokers/ids/<id> and the owner session matches neither the current session nor the broker’s last-recorded session, the broker’s retry logic has given up. The first session’s create succeeded but the response was lost, the broker never recorded that session as its own, and on reconnect it sees a NodeExists whose owner it does not recognize. The ephemeral persists until the orphan session times out, but the broker will not retry on its own.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
zk_ephemerals_countTracks ephemeral node count; orphans inflate itFlat or growing while clients report NodeExists suggests orphans
zk_num_alive_connectionsValidates whether a reconnect storm is underwaySharp drop then spike is the session-expiry V-shape
zk_outstanding_requestsPipeline saturation produces client timeouts that masquerade as ConnectionLossSustained non-zero under load
zk_server_stateConfirms whether the ensemble is mid-electionFlapping between leader and looking
zk_fsynctime (3.6+)Slow writes cause client timeoutsSustained elevation during the NodeExists burst

Fixes

ConnectionLoss retry race (the common case)

This is almost always a client-code issue. The fix is to treat ConnectionLoss as “outcome unknown,” not “operation failed.” After reconnecting, the client should exists() or getData() on the target path before retrying create(). If the node is there and (for ephemerals) owned by the current session, treat the original create as successful.

For non-idempotent creates where the caller cannot tell whether it owns the resulting node, use a sequential node (CreateMode.PERSISTENT_SEQUENTIAL or EPHEMERAL_SEQUENTIAL) and have the caller record the full path it intends to claim. Sequential nodes never conflict on retry because each call produces a unique path. This is the standard ZooKeeper lock-recipe pattern.

Orphaned ephemeral blocking re-registration

If a legitimate process cannot re-register because an orphaned ephemeral from a previous session is still present, two options, in order of preference:

  1. Wait. If the orphaned session has expired, the leader’s expiry sweep will reap the node. This usually completes within the session-timeout window.
  2. Delete the znode manually. Destructive. Deleting an ephemeral that another process legitimately owns will cause that process to lose its registration. Confirm the owning session is gone from dump output first. In zkCli.sh:
    delete /some/path
    For Kafka broker registration, the common operator fix is to delete /brokers/ids/<id> and restart the broker. Clearing log directories is frequently suggested in operator threads but is a blunt instrument that risks data loss and does not address the root cause.

Genuine concurrent race (leader election / distributed lock)

Not a bug. Two candidates tried to create the same node; one won, one lost. The loser should watch the node and either back off or retry when it disappears. If the application code does not handle NodeExists in its leader-election path, that is the bug.

If the rate of these races is high, the cause is usually too many candidates starting simultaneously (deployment stampede) or too-short election timeouts causing repeated re-election. Check the application’s startup pattern and the ensemble’s zk_server_state stability.

Persistent node from a previous deploy

Confirm with get /some/path that ephemeralOwner is 0 (persistent), verify no current process expects to own it, then delete it. The next create() will succeed. This is the only case where the fix is purely operational: persistent nodes never expire.

Prevention

  • Treat ConnectionLoss as an unknown outcome. Every ZooKeeper client should have a recovery path that checks state (exists, getData) before retrying non-idempotent operations. Put this in the client library, not in individual callers.
  • Prefer sequential nodes for any create that may be retried. Sequential creates never conflict because each produces a unique path. The caller records the path and reuses it on retry.
  • For ephemeral nodes, store the session ID you used. When a retry returns NodeExists, compare ephemeralOwner to your stored session ID and to your current session ID. If either matches, you own the node.
  • Do not panic-delete on NodeExists. A naive “delete then recreate” loop can destroy a peer’s legitimate node. Always check ownership first.
  • Monitor connection stability. Most retry storms originate upstream of the NodeExists, in connection drops and leader elections. The NodeExists rate is a symptom; the cause is in the connection layer.
  • For Kafka on ZooKeeper mode, plan the migration to KRaft. KRaft removes ZooKeeper entirely and eliminates this class of error. KRaft is production-ready for new clusters in Kafka 3.3+.

How Netdata helps

  • Per-second collection of zk_ephemerals_count and zk_num_alive_connections lets you correlate a NodeExists burst with the connection-loss event that caused it, instead of treating each as an independent incident.
  • The ZooKeeper overview dashboard correlates ephemeral count drops with connection count drops, making the session-expiry V-shape visible at a glance.
  • Per-host latency histograms (zk_fsynctime on 3.6+) let you confirm or rule out write-path stalls as the upstream cause. If writes are fast and zk_server_state is stable, the NodeExists is purely a client-side retry issue.
  • Leader-only metrics (zk_synced_followers, zk_pending_syncs) are collected from whichever node currently reports leader, so replication-driven reconnect storms that manifest as NodeExists in client logs are not missed.