ZooKeeper KeeperErrorCode = ConnectionLoss: the transient disconnect every client hits

KeeperErrorCode = ConnectionLoss is the error every ZooKeeper client eventually logs. It means the TCP connection between the client and the server it was talking to broke before the operation’s response arrived. It does not mean the operation failed, and it does not mean the session is gone. The outcome of the in-flight operation is unknown, and the correct response is an idempotent retry.

The most common confusion is treating ConnectionLoss like SessionExpired. They are different things. ConnectionLoss is a transient client-side condition: the TCP link is down but the session may still be alive on the ensemble. SessionExpired is an ensemble-level verdict: the cluster has declared the session dead and deleted its ephemerals. The cluster owns the expiry decision, and the client only learns about it after it reconnects.

A single ConnectionLoss from one client is almost always benign: a server restart, a network blip, a load-balancer reset, or a client-side GC pause. A burst of ConnectionLoss across many clients at the same instant is the signature of a server-side pause, a leader election, or a partition. The first job during an incident is to figure out which of those you are looking at.

What this means

When the client library sees the TCP socket break, any operation in flight returns KeeperErrorCode = ConnectionLoss to the application. The library does not know whether the request reached the server, whether it was applied, or whether the response was lost on the way back. For writes this matters: a create may have succeeded on the ensemble even though the client saw ConnectionLoss.

stateDiagram-v2
    [*] --> Connected: session negotiated
    Connected --> Disconnected: TCP breaks, in-flight op returns ConnectionLoss
    Disconnected --> Connected: reconnect succeeds within session timeout
    Disconnected --> Expired: ensemble declares session expired
    Expired --> [*]: ephemerals deleted, must create a new session

Key properties:

• The session may still be alive. Sessions are tracked by the ensemble, not the client. As long as the client reconnects to any ensemble member within the negotiated session timeout, the session and its ephemerals, watches, and ACL state survive.
• The client library reconnects on its own. Do not tear down and rebuild the ZooKeeper handle on ConnectionLoss. The library retries the connection string. Only construct a new session after the library signals that the session is expired.
• The operation outcome is unknown. The application must reconcile state, re-read, or use a version check, rather than blindly re-submitting.
• ZooKeeper does not auto-retry non-idempotent operations. The client library preserves ordering guarantees; the application owns the retry semantics.

Common causes

Quick checks

# Confirm four-letter commands are whitelisted (3.5.3+) and the process is alive
echo ruok | nc localhost 2181
echo mntr | nc localhost 2181 | head -5

# Is the node up and read-write?  Returns "rw" or "ro".
echo isro | nc localhost 2181

# Server role and uptime
echo mntr | nc localhost 2181 | grep -E 'zk_server_state|zk_uptime'

# Recent election activity
echo mntr | nc localhost 2181 | grep -E 'zk_looking_count|zk_.*leader_unavailable_time'

# GC pause impact (3.6+)
echo mntr | nc localhost 2181 | grep -E 'zk_.*jvm_pause'

# Write-path health
echo mntr | nc localhost 2181 | grep -E 'zk_.*fsynctime|zk_outstanding_requests|zk_throttled_ops'

# Per-IP connection rejections and totals
echo mntr | nc localhost 2181 | grep -E 'zk_connection_rejected|zk_num_alive_connections'

# Per-source-IP connection distribution (expensive on loaded servers, sample sparingly)
echo cons | nc localhost 2181 | awk '{print $2}' | cut -d: -f1 | sort | uniq -c | sort -rn

These commands are read-only against ZooKeeper. They still require 4lw.commands.whitelist to include them since 3.5.3. The cons command is O(connections) and can cause a latency spike on busy servers; do not run it on a tight loop in production.

How to diagnose it

1. Scope the blast radius first. Is this one client, one client IP, one AZ, or the whole fleet? One client points at the client or the network path. Whole fleet points at a server-side event. Pull client-side logs and group by source.
2. Confirm whether sessions actually expired. ConnectionLoss alone does not delete ephemerals. Check zk_stale_sessions_expired, zk_ephemerals_count, and zk_connection_drop_count deltas. If those did not move and ephemerals did not drop, sessions survived and the application’s retry logic is the only remediation needed.
3. Check for an election. zk_looking_count increments, zk_server_state shows “looking” briefly, and zk_sum_leader_unavailable_time grows. Any of those point at a leader re-election. The next question is why the leader went down.
4. Check for a JVM pause. zk_jvm_pause_time_ms p99 spiking is the signature of a Stop-the-World GC. If the leader paused long enough to miss heartbeats, followers triggered an election. GC is the most common cause of unnecessary elections. Cross-reference with the GC log.
5. Check for a disk stall. zk_fsynctime p99 elevated, zk_outstanding_requests climbing, zk_throttled_ops incrementing. This is the write-pipeline deadlock pattern. Reads on followers may stay fast and hide the severity.
6. Check for client-side patterns. Fixed-interval ConnectionLoss at, say, 60s or 300s points at an LB or NAT idle timeout shorter than the client’s ping interval (the library pings after half the negotiated session timeout of idle time). Single-IP clusters point at maxClientCnxns. Confirm with zk_connection_rejected.
7. Check the client library version. On 3.9.3 client libraries, the ZOOKEEPER-4921 reconnect regression means a single brief network blip expires the session instead of recovering .

Metrics and signals to monitor

Fixes

Single-client ConnectionLoss

Retry idempotently. The session is most likely still alive. For writes, reconcile by re-reading and using a version check (setData with the correct version, or create with a deterministic path) rather than blindly re-submitting. If you use Apache Curator, the framework’s retry policy already wraps operations. Tune RetryPolicy to your workload and decide whether SessionConnectionStateErrorPolicy should treat SUSPENDED as an error (the default, conservative) or only LOST.

Leader re-election

If the leader went down because of a deployment, document the expected impact and tighten the rolling-restart procedure. If the election was unplanned, find the root cause before the next one: elections are not normal steady-state behavior. The usual triggers are GC pauses and fsync stalls, both of which are detectable before they cause an election.

JVM GC pauses

Increase heap if the data tree genuinely needs it, but be careful: bigger heaps mean longer full GC pauses. Prefer G1GC or ZGC (JDK 15+) for low-pause behavior . Track zk_znode_count and zk_watch_count; unbounded growth is the usual root cause of escalating GC pressure. Disable Transparent Huge Pages on the host; THP compaction can extend GC pauses significantly .

Transaction-log fsync stall

Put dataLogDir on a dedicated low-latency device. Never colocate the txnlog with snapshots or with other workloads. On cloud instances, watch for burst-credit exhaustion on EBS gp2 or gp3; the transition from burst to baseline is abrupt. If the stall is real and persistent, the disk is undersized for the write rate.

Load-balancer or NAT idle timeouts

Either raise the idle timeout above the client’s ping interval, or remove the LB from the ZooKeeper client path. ZooKeeper clients should connect directly to ensemble members; an L4 LB in the path breaks the session-affinity assumptions the client library depends on.

maxClientCnxns rejection

Raise the per-source-IP limit, or stop NAT’ing many pods through one host IP. The default of 60 is appropriate for a handful of long-lived clients, not for a fleet of microservices sharing a host.

Large requests

Reduce znode size. ZooKeeper is a coordination service, not a database. If getChildren on a parent with tens of thousands of children returns ConnectionLoss, restructure the tree or move the data out of ZooKeeper. Raising jute.maxbuffer (default 1MB) treats the symptom, not the cause, and increases heap pressure.

3.9.3 reconnect regression

Upgrade client libraries to 3.9.4 or later .

Prevention

• Retry idempotently in every client. Treat ConnectionLoss as “unknown outcome”, not “failure”. Use version checks or deterministic keys for writes.
• Monitor elections. Any unplanned zk_looking_count increment is a ticket, not background noise.
• Monitor fsync and GC. Both are detectable before they trigger elections or session storms.
• Size heap to the data tree, not to a default. Track zk_znode_count and zk_approximate_data_size over time.
• Keep dataLogDir on dedicated storage. This is the single highest-impact configuration change for write stability.
• Pin client library versions and track CVEs. CVE-2023-44981 (SASL auth bypass) is fixed in 3.9.1, 3.8.3, and 3.7.2 . CVE-2024-6763 (Jetty in the admin server) .
• Whitelist four-letter commands deliberately. Since 3.5.3, commands like mntr and ruok must be in 4lw.commands.whitelist. A missing whitelist silently returns empty output that monitoring may misread as “all zeros healthy”.

How Netdata helps

• Correlate zk_looking_count and zk_sum_leader_unavailable_time against per-second JVM pause metrics to confirm whether an election was GC-induced, without grepping logs.
• Track zk_fsynctime p99 alongside OS-level disk metrics (iowait, await) on the txnlog device to localize write stalls to the storage layer.
• Watch zk_num_alive_connections, zk_connection_drop_count, and zk_stale_sessions_expired together to distinguish ConnectionLoss from actual session expiry in real time.
• Surface zk_connection_rejected and zk_large_requests_rejected as leading indicators before clients complain about “random” disconnects.
• Per-second collection catches the burst pattern of an election or GC pause that minute-level scraping misses.

Related guides

• ZooKeeper data size growing: using ZooKeeper as a database is an anti-pattern
• ZooKeeper autopurge not configured: snapshots and logs filling the disk over months
• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “Client session timed out, have not heard from server”: the heartbeat miss
• ZooKeeper connection drops spiking: sessions dying in bursts
• ZooKeeper dataLogDir sharing a disk with snapshots: the #1 fsync-latency footgun
• ZooKeeper “Detected pause in JVM or host machine (eg GC)”: the pause-monitor warning
• ZooKeeper transaction log disk full: the crash with no graceful degradation
• ZooKeeper follower doing a SNAP sync: full snapshot transfer and its blast radius
• ZooKeeper follower sync time climbing: a follower approaching ejection
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls