ZooKeeper session count climbing: leaks and duplicate sessions

zk_global_sessions climbing while your client fleet is stable is a slow ZooKeeper failure mode. The ensemble keeps serving reads and writes, latency looks fine, quorum is intact, but the session table keeps growing. Each entry costs heap and periodic heartbeat processing. Eventually you hit a GC death spiral, an OOM, or a maxClientCnxns-shaped outage.

The signal is simple to read but easy to misinterpret. ZooKeeper exposes two distinct populations: global sessions, which the leader echoes across the ensemble, and local sessions (only present when localSessionsEnabled=true, added in ZK 3.5, default false), which live on a single follower and upgrade to global when the client creates an ephemeral node. If you only watch zk_global_sessions you may be looking at a subset of the actual session population, and if you only watch zk_num_alive_connections you cannot tell a leak from a deployment event.

The three signals that matter are zk_global_sessions, zk_num_alive_connections, and zk_ephemerals_count.

What this means

A global session in ZooKeeper is the unit of client identity that the leader has agreed to track across the ensemble. Ephemeral nodes, watches, and ACL enforcement all bind to a session ID. The leader maintains a session tracker with expiry buckets; each session must renew within its negotiated timeout or it expires, taking its ephemeral nodes and watches with it.

zk_global_sessions from mntr reports the count of these leader-tracked sessions. Each session consumes a small but non-zero amount of heap (session state, watch table entries if the client sets watches, queued request state) and requires periodic heartbeat processing on the request pipeline.

Monotonic growth in zk_global_sessions that never plateaus, without a matching change in client deployment, is the canonical tell for one of two pathologies.

1. Session leak: a client library or application opens new sessions without retiring old ones. The leak is usually client-side (a framework that spawns new ZooKeeper handles without closing the previous one), less often server-side (a buggy reconnection path that establishes a new session before the old one expires).

2. Duplicate sessions from reconnection churn: clients enter a reconnect loop where each cycle creates a new session alongside the previous one, either because the client never observes the SESSION_EXPIRED event that should gate new session creation, or because a framework’s reset path races the user code’s close path.

The third pattern worth knowing is benign growth from local-session promotion. With localSessionsEnabled=true, sessions start local to a follower and are upgraded to global when they create ephemeral nodes. A workload shift that increases ephemeral-node creation will inflate zk_global_sessions without any change in actual client count.

flowchart TD
    A[zk_global_sessions climbing] --> B{num_alive_connections also climbing?}
    B -- Yes, proportional --> C[Real client growth, check deployment]
    B -- No, flat or declining --> D{ephemerals_count also climbing?}
    D -- Yes, proportional --> E[Legitimate ephemeral use or local-session upgrade]
    D -- No --> F[Session leak or duplicate-session churn]
    F --> G{stale_sessions_expired climbing?}
    G -- Yes --> H[Reconnection loop, investigate client libraries]
    G -- No --> I[Pure leak, clients opening new handles without closing]

Common causes

Quick checks

All read-only and safe in production. cons and dump are O(n) over the connection and session tables and can cause brief latency spikes on large ensembles; sample sparingly.

# Headline session count
echo mntr | nc localhost 2181 | grep zk_global_sessions

# Pair with live connection count and ephemeral count
echo mntr | nc localhost 2181 | grep -E 'zk_(num_alive_connections|ephemerals_count)'

# Session expiration and connection drop rates
echo mntr | nc localhost 2181 | grep -E 'zk_(stale_sessions_expired|connection_drop_count)'

# Per-connection detail, including session IDs (sid=0x...). Expensive, sample sparingly
echo cons | nc localhost 2181

# Outstanding sessions and ephemeral nodes. Expensive, sample sparingly
echo dump | nc localhost 2181

# Check whether local sessions are enabled (changes interpretation of zk_global_sessions).
# Path varies by distribution.
grep -E 'localSessionsEnabled|localSessionsUpgradingEnabled' /etc/zookeeper/conf/zoo.cfg

# JVM pause times. A leading cause of session expiration cascades.
echo mntr | nc localhost 2181 | grep -i jvm_pause
<!-- TODO: verify the exact JVM pause metric name(s) exposed by your ZK version; names and availability vary -->

# Per-IP connection limit and rejected-connection counter
echo mntr | nc localhost 2181 | grep connection_rejected
grep -E 'maxClientCnxns' /etc/zookeeper/conf/zoo.cfg

If mntr returns nothing, your four-letter words may not be whitelisted. Since ZK 3.5.3 you must set 4lw.commands.whitelist (or the Java system property zookeeper.4lw.commands.whitelist) to include at least mntr. The AdminServer on port 8080 is the long-term replacement for four-letter words.

How to diagnose it

1. Establish the baseline. Capture zk_global_sessions, zk_num_alive_connections, and zk_ephemerals_count at one-minute intervals for at least 30 minutes. The single most important question is whether the growth is monotonic (never plateaus, never reverses) or stepwise (jumps, then flat). Pure monotonic growth without any plateau is the leak signature.

2. Compute the session-to-connection ratio. In steady state, zk_global_sessions / zk_num_alive_connections should be roughly constant. If the ratio is climbing, you have more sessions than connections can explain, which points at duplicate sessions or sessions that outlive their connections. If the ratio is constant and both are climbing together, look for a real client growth event first.

3. Check whether local sessions are in play. If localSessionsEnabled=true, zk_global_sessions only counts sessions promoted to global (typically by creating an ephemeral node). A workload shift that increases ephemeral creation will inflate zk_global_sessions without any leak. Confirm by checking whether zk_ephemerals_count is climbing proportionally.

4. Pull per-connection detail and look for duplicate session IDs from the same source IP. Use cons to list every connection with its session ID. Multiple live sessions originating from the same client IP and process indicate a duplicate-session bug, not a generic leak.

5. Correlate with session expirations. If zk_stale_sessions_expired is also climbing, the leak is dynamic: sessions are expiring but new ones are being created faster. If zk_stale_sessions_expired is flat while zk_global_sessions climbs, sessions are accumulating without expiring, which is a harder leak where the server believes the sessions are alive even though the clients have moved on.

6. Check JVM pause times on both server and client. Server-side GC pauses cause session expiration cascades that masquerade as churn. Client-side GC pauses cause clients to miss heartbeats and reconnect, often creating transient duplicate sessions. The server-side signal is the JVM pause metric from mntr; the client-side signal must be read from the client’s own GC logs.

7. Inspect the client library. The single most common cause of duplicate ZooKeeper sessions in production is a client framework bug, not a ZooKeeper bug. Apache Curator has a known class of issues where the framework’s reset path can race the user’s close path on session expiration, leaving an orphaned session connected alongside the new one. Check the framework version against the upstream issue tracker before changing anything on the server.

Metrics and signals to monitor

Fixes

Client framework session leak

The right fix is in the client, not the server. For Apache Curator specifically, upgrade to the latest patch release and audit the application’s CuratorListener implementation. The mandatory rule from the ZooKeeper FAQ is: only create a new session when you are notified of SESSION_EXPIRED, and SESSION_EXPIRED automatically closes the existing ZooKeeper handle. Application code that opens a new handle on any connection-state change (not just expiration) will leak sessions.

If you cannot immediately push a client fix, you can buy time by raising the client-side session timeout so fewer expirations occur per unit time. This slows the leak but does not stop it. A scheduled bounce of the leaking client process, keeping the session count below your heap headroom threshold, is a stopgap, not a fix.

Do not bounce the ZooKeeper ensemble to clear leaked sessions. The clients will reconnect and recreate the leak within minutes, and you will have paid an availability cost for nothing.

Duplicate sessions from reconnection churn

Treat this as a client bug with server-side symptoms. The diagnostic question is: why is the client creating a new session before the old one is observed as expired? Common root causes.

• Client-side JVM GC pauses longer than the negotiated session timeout. Fix the client heap, switch to G1GC (JVM default since JDK 9) or ZGC (production-ready since JDK 15), and consider raising the session timeout if the client legitimately needs long GC pauses.

• Network instability between the client and ensemble that causes repeated TCP resets. Stabilize the network path or move the client closer to the ensemble.
• A framework’s reconnect logic that does not wait for the SESSION_EXPIRED callback before establishing a new session.

Local-to-global session promotion misread as a leak

If localSessionsEnabled=true and your workload is creating more ephemeral nodes than before, zk_global_sessions will climb because local sessions are being promoted. This is benign and expected. Confirm by checking that zk_ephemerals_count is climbing proportionally, that zk_num_alive_connections is stable, and that no workload change increased ephemeral-node creation unexpectedly.

No fix is required. If the promotion volume is operationally problematic, reconsider whether the ephemeral-creation workload belongs on ZooKeeper at all.

maxClientCnxns saturation

If zk_connection_rejected is non-zero, clients are being silently denied service because their source IP has hit the per-IP connection limit. This is not strictly a session leak, but it produces symptoms (clients cannot connect, applications spawn new connection attempts that also fail) that look like one. Raise maxClientCnxns in zoo.cfg (default 60 per source IP; set to 0 to remove the limit) and rolling-restart. In containerized environments where many pods share a host IP, the default 60 is easily exceeded.

Prevention

• Track the session-to-connection ratio over time. It should be stable. Alert on sustained drift, not on absolute session count.
• Cap and monitor client-side connection pools. Each ZooKeeper handle is a session. Frameworks that let application code spawn handles freely will eventually leak.
• Pin client framework versions and watch their issue trackers. Curator and similar frameworks have shipped session-leak bugs more than once.
• Enable GC logging on ZooKeeper clients, not just the server. Client-side GC pauses are a leading root cause of session churn.
• Run a periodic chaos exercise that disconnects a fraction of clients and watches the ensemble’s session table return to baseline. If it does not return, you have a leak; find it before it finds you at 3 a.m.
• Set maxClientCnxns deliberately based on your container-to-host-IP fan-in ratio. The default 60 is too low for most microservice deployments that fan many pods through one host IP.

How Netdata helps

• Per-second collection of zk_global_sessions, zk_num_alive_connections, and zk_ephemerals_count makes the monotonic-growth signature visible within minutes rather than after a daily aggregation rolls over.
• ML anomaly detection flags session-to-connection ratio drift even when both metrics are individually within normal absolute ranges.
• Correlating zk_global_sessions with JVM pause, zk_stale_sessions_expired, and zk_connection_drop_count in a single view distinguishes a server-side GC cascade from a client-side reconnect loop without manual cross-referencing.
• Composite dashboards surface leader-only metrics (zk_followers, zk_synced_followers) alongside connection metrics, so quorum degradation can be ruled out as a confounding factor.
• The same per-second resolution applies to host-level signals (FD usage, disk latency, network retransmits) that often explain why a client is reconnecting in the first place.

Related guides

• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “Client session timed out, have not heard from server”: the heartbeat miss
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper leader election storm: an ensemble that keeps re-electing
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper outstanding requests growing: the request pipeline is backing up
• ZooKeeper proposals not committing: proposal_count outpacing commit_count
• ZooKeeper quorum ack latency high: followers slow to acknowledge proposals
• ZooKeeper quorum loss: no leader elected and every write is failing