ZooKeeper KeeperErrorCode = Session expired: ephemeral nodes gone, clients evicted

The exact error clients log is KeeperErrorCode = Session expired. On the ZooKeeper side you see Expiring session 0x... timeout of Nms exceeded. Once that line lands, the client’s ZooKeeper handle is dead and every piece of state it owned through that session is gone: ephemeral znodes deleted, watches invalidated, ACLs no longer enforceable. The client cannot reconnect on the same handle. It must build a new ZooKeeper object, negotiate a new session, and recreate every ephemeral node it relied on.

Session expiration is server-managed, not client-managed. The leader’s session tracker holds every session in expiry buckets. If the cluster does not hear from a client inside the negotiated timeout window, the leader marks the session expired and the deletion cascade starts. Restarting the client alone never rescues the session: the server decided it was dead, and the contract is final.

The blast radius is usually much larger than the one expired session. Ephemeral nodes are how Kafka brokers register, how HBase RegionServers signal liveness, and how distributed lock holders prove ownership. When those nodes vanish, watches fire across every consumer watching them, and downstream systems react in lockstep: broker deregistration, region reassignment, lock handoff (or, in the worst case, dual lock holders). What looked like a single bad client is suddenly a multi-system incident.

What this means

When the leader expires a session:

1. All ephemeral nodes owned by that session are deleted from the data tree.
2. Every watch registered on those znodes fires, producing a notification burst proportional to watch fan-out.
3. The client’s ZooKeeper handle becomes invalid. Any subsequent operation on that handle throws. The client library must construct a new ZooKeeper object, establish a new session, and re-register all ephemeral state.
4. Dependent systems observe the ephemeral deletions and act on them: Kafka sees broker IDs disappear, HBase sees RegionServer znodes vanish, lock frameworks see the lock node go.

The client library sends heartbeats at roughly one-third of the negotiated session timeout and reconnects to another ensemble member if it does not see a response in another third. If the client cannot complete heartbeats across the full timeout, the cluster expires the session. An expiration is always a story about what stopped heartbeats from landing inside the window: the client stopped sending (GC, SIGSTOP, swap), the server stopped processing (GC, fsync stall, request pipeline saturation), or the network stopped delivering (partition, dropped packets, asymmetric routing).

flowchart TD
    A[Trigger: client GC, server GC, network blip, leader failover] --> B[Heartbeats miss for full session timeout]
    B --> C[Leader expires session]
    C --> D[Ephemeral nodes deleted]
    C --> E[Watches fire]
    D --> F[Downstream reacts]
    E --> F
    F --> F1[Kafka broker deregistered]
    F --> F2[HBase regions reassigned]
    F --> F3[Distributed lock lost or split]
    C --> G[Client must create new ZK handle]
    G --> H[Recreate ephemerals, re-register watches]
    H --> I[Reconnection storm]

A few bounds worth remembering when you read the logs:

• minSessionTimeout defaults to 2 * tickTime (4000ms at the default tickTime=2000).
• maxSessionTimeout defaults to 20 * tickTime (40000ms).
• If a client requests a timeout outside those bounds, the server clamps it silently. Kafka operators setting zookeeper.session.timeout.ms=1000 may actually be running with 4000ms and never know.
• Cloud and virtualized environments have higher and more variable latency. A 5-second session timeout is too aggressive for EC2-grade hosts.

Common causes

Quick checks

Run these read-only checks in the first five minutes. None of them mutate state.

Note: on ZooKeeper 3.5+, the four-letter-word commands (ruok, isro, mntr) are disabled by default. You must set 4lw.commands.whitelist=ruok,isro,mntr,stat in zoo.cfg and restart, or these commands return nothing.

# Server process alive and command port listening?
echo ruok | nc localhost 2181

# Read-write or read-only? "ro" means quorum lost, writes failing.
echo isro | nc localhost 2181

# Confirm exactly one leader in the ensemble and see what this node reports.
echo mntr | nc localhost 2181 | grep zk_server_state

# Direct counter of expired sessions.
# <!-- TODO: verify: zk_stale_sessions_expired is not in upstream mntr. May require JMX exporter or custom build. -->
echo mntr | nc localhost 2181 | grep zk_stale_sessions_expired

# Mass disconnect signature (standard mntr metrics).
echo mntr | nc localhost 2181 | grep -E 'zk_num_alive_connections|zk_ephemerals_count'

# Watch fan-out during the cascade (standard mntr metrics).
echo mntr | nc localhost 2181 | grep -E 'zk_packets_received|zk_packets_sent|zk_watch_count'

# <!-- TODO: verify: jvm_pause metrics are not in upstream mntr. Check JMX or your monitoring system instead. -->
echo mntr | nc localhost 2181 | grep jvm_pause

# <!-- TODO: verify: zk_looking_count and leader_unavailable_time are not in upstream mntr. Check JMX or election logs. -->
echo mntr | nc localhost 2181 | grep -E 'zk_looking_count|zk_.*leader_unavailable_time'

# <!-- TODO: verify: fsynctime metrics are not in upstream mntr. Check JMX or disk-level instrumentation. -->
echo mntr | nc localhost 2181 | grep -E 'zk_.*fsynctime|zk_outstanding_requests'

# Server-side confirmation of the expiration.
grep -E "Expiring session|Session expired" /var/log/zookeeper/zookeeper.log | tail -50

On a healthy node these counters are quiet. A burst in session expirations paired with a drop in zk_num_alive_connections and zk_ephemerals_count is the fingerprint of a session expiration storm.

How to diagnose it

1. Scope it first. Is this one client, one application fleet, or every client of the ensemble? One isolated session expiring is almost always a client-side GC pause or a stuck client process. Many sessions expiring across unrelated systems points at the server or the network.
2. Verify the ensemble is functional. ruok returns imok even when the node is in LOOKING state, so do not stop there. Use isro (must be rw) and zk_server_state via mntr (exactly one leader, the rest followers). If the ensemble has lost quorum, you have a different incident; see ZooKeeper quorum loss: no leader elected and every write is failing.
3. Correlate with leader elections. A leader election in the same minute as the expirations tells you the session churn was a side effect of a failover. Investigate the failover itself, not the expirations.
4. Look at the leader’s JVM. A GC pause longer than tickTime (default 2000ms) is in the danger zone for followers; a pause approaching minSessionTimeout will start expiring client sessions directly.
5. Look at fsync time on the leader. Long fsync on the leader does not just stall writes. With ZOOKEEPER-1740 class behaviour, when the leader finally resumes after a long fsync stall, it expires a batch of sessions whose timeout counters were not paused during the stall. Check the transaction log device latency and host-level iowait. If fsync is the cause, the fix is disk-side, not session-side.
6. Check the network. If neither GC nor fsync explains the timing, look at packet counters. zk_packets_received should track your client population. A sudden drop while clients are known to be active is a connectivity event between clients and the ensemble, even if the ensemble looks healthy from inside.
7. Inspect the client side. On the affected client, look at the JVM GC log and the application’s session event listener. Curator-based clients log state transitions (SUSPENDED, RECONNECTED, LOST). The transition from SUSPENDED to LOST is the moment the session actually died from the client’s perspective.
8. Watch for the dual-lock-holder trap. If a client holds a distributed lock backed by an ephemeral node and its JVM pauses longer than the session timeout, the ephemeral node is deleted, another client acquires the lock, and the paused client wakes up still believing it holds the lock. Two holders, no error message. This is Curator Tech Note 10. If your incident looks like a split lock, this is the cause.

Metrics and signals to monitor

Server-side metrics alone miss half the picture. Instrument the client library too: log every state transition and every reconnect attempt. The gap between “server looks healthy” and “clients are losing sessions” is where most production pain lives.

Fixes

Client-side GC pause

If the client’s JVM froze longer than the remaining session window, the server had no choice but to expire the session. Tune the client GC so that stop-the-world pauses are a small fraction of the session timeout, and raise the client’s requested session timeout above the worst realistic GC pause. Curator’s default session timeout is 60 seconds for exactly this reason. ZGC (JDK 15+) reduces pause times to single-digit milliseconds and largely removes this failure mode.

Server-side GC pause

If server-side JVM pause metrics show the spikes, the fix is heap and GC. Check whether the data tree has grown to drive the pressure: pull zk_znode_count, zk_approximate_data_size, and zk_watch_count via mntr. A growing tree will eventually produce a GC death spiral regardless of GC algorithm. Increase heap, switch to ZGC if you are on a supported JDK, and address the underlying growth. If a tree cleanup is required, be careful: deleting znodes fires watches and can amplify the incident.

Network blip

If the ensemble is fine and clients lost reachability, fix the path. Common causes: firewall or security group change, asymmetric routing, MTU mismatch causing fragmentation and drops, switch firmware issues, DNS failures. The ZooKeeper troubleshooting wiki is explicit that virtualized environments have higher and more variable latency and that 5-second timeouts are too low. Raise session timeouts for cloud deployments and verify the path with isro, TCP reachability, and mtr between client fleet and ensemble.

Leader failover

If the expirations line up with a leader election, the failover is the incident and the expirations are a symptom. The leader became unresponsive: GC pause, disk stall, or network partition between the leader and a quorum of followers. Investigate the failover itself. See ZooKeeper leader election storm: an ensemble that keeps re-electing and ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls.

Long fsync on the leader

If fsync latency is elevated, treat this as a disk incident. The first-line fix is structural: put the transaction log on its own dedicated device (dataLogDir), separate from snapshots. In cloud environments, check whether EBS burst credits or provisioned IOPS are exhausted. Do not paper over fsync latency with longer session timeouts; you will mask the underlying failure until it cascades.

Stale ephemeral node blocking re-registration

In a subset of cases the client expires, restarts, and cannot re-register because the old ephemeral node is still present. The new session tries to create the same path, sees it exists, and fails. This is a known class of bug (for example, ZOOKEEPER-4837 and the older ZOOKEEPER-2919 ). The workaround is to delete the stale node manually, then have the client recreate it under the new session. Capture the znode path, owner session ID, and the version so the delete is unambiguous.

Prevention

• Right-size session timeouts for the environment. Defaults are too tight for cloud and virtualized hosts. Use at least 10 to 20 seconds for cloud deployments and confirm the server does not clamp the request silently.
• Tune GC on every JVM that talks to ZooKeeper, client and server. Stop-the-world pauses shorter than one-third of the session timeout is a reasonable target. ZGC where available.
• Separate transaction log disk. dataLogDir on its own device is the single highest-leverage config change for write-path stability.
• Alert on leader elections. Every unplanned election should be a ticket at minimum. Suppress during planned maintenance, then enforce.
• Instrument client session events. The client library sees problems the server never will. Log every SUSPENDED, RECONNECTED, and LOST transition with a timestamp.
• Run controlled failover tests. Killing a ZooKeeper node in staging or a canary production cell is the only way to confirm both the system and the monitoring behave. Many teams first discover broken client reconnect logic during a real incident.
• Watch the data tree. zk_znode_count and zk_approximate_data_size should be trended and capacity-planned, not just checked. Heap exhaustion from tree growth produces the GC cascade that produces the session expiration storm.

How Netdata helps

• Per-second collection on zk_num_alive_connections, zk_ephemerals_count, and zk_packets_received makes the V-shape of a session expiration storm immediately visible instead of smoothed away by a 60-second scrape interval.
• Correlating JVM pause metrics against zk_outstanding_requests and connection counts in a single pane separates a server-side GC cascade from a client-side pause, without pivoting between dashboards.
• ML anomaly detection flags synchronized drops in connection count and ephemeral count even when absolute values stay inside static thresholds, which is the signature of an early-stage storm.
• Per-leader views of fsync latency, election events, and leader unavailability time let you pin the root cause to disk, election, or network in the first minute instead of the first hour.
• The same metrics stream surfaces follow-on signals (zk_packets_sent watch-fan-out spikes, connection drop rates) that tell you whether downstream systems are about to be hit by the cascade.
• Anomaly-aware alerting on zk_server_state distinguishes planned rolling-restart elections from unplanned failover without per-window suppression churn.

Related guides

• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper leader election storm: an ensemble that keeps re-electing
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper outstanding requests growing: the request pipeline is backing up
• ZooKeeper quorum ack latency high: followers slow to acknowledge proposals
• ZooKeeper quorum loss: no leader elected and every write is failing
• ZooKeeper read latency high: memory reads that should never be slow
• ZooKeeper server stuck in LOOKING: a node that never rejoins the quorum