Consul session invalidation: distributed locks releasing across the cluster

Sessions are Consul’s mechanism for binding distributed locks to node and application health. When a session is invalidated, every KV key it holds is acted on according to its Behavior. With the default release, the lock holder is cleared and the key’s ModifyIndex increments. With delete, the key is removed entirely. Every consumer watching that key reacts.

A spike in session invalidation surfaces as a cluster-wide release of distributed locks. Leader elections fire, caches flush, configuration reloads trigger, and any logic keyed off ModifyIndex change wakes up. The operator’s job is to identify which sessions invalidated, why, and whether the root cause is a single node, a renewal-path failure, or a gossip flap.

What this means

A Consul session is an ephemeral contract that ties held KV keys to liveness signals: a TTL, one or more health checks (by default serfHealth), or both. Sessions back distributed locks, leader election, and coordinated configuration. When the contract breaks, the session is invalidated and everything it held is released or deleted.

Invalidation triggers:

• Node deregistered or failed: all sessions bound to that node invalidate at once.
• A bound health check goes critical or is deregistered: any session tied to that check invalidates. With the default serfHealth, a gossip flap is enough.
• TTL expires: the session was not renewed before its TTL elapsed.
• Explicit destroy: PUT /v1/session/destroy/:uuid is called. The endpoint is idempotent; destroying a non-existent or already-expired session returns 200.

On invalidation, the Behavior field decides what happens to each held key. release (the default) clears the lock and bumps ModifyIndex so watchers see a change. delete removes the key entirely. After invalidation, Consul enforces a LockDelay that blocks any contender from re-acquiring the previously held lock for that interval. The default is 15 seconds; valid range is 0 to 60 seconds.

Two properties shape the blast radius. Sessions are not reentrant: an application that loses its session and tries to re-acquire the same lock may find another instance already holds it. And a single invalidation can release many keys at once if one session held multiple locks.

flowchart TD
    A[Trigger: node failure, TTL expiry, gossip flap] --> B[Session invalidated]
    B --> C{Session Behavior}
    C -->|release| D[Lock holder cleared, ModifyIndex bumped]
    C -->|delete| E[KV key removed entirely]
    D --> F[LockDelay blocks re-acquire]
    E --> F
    F --> G[App reacts: leader election, cache flush, reload]

Common causes

Quick checks

Read-only and safe during an incident.

# List all active sessions, grouped by node
curl -s http://127.0.0.1:8500/v1/session/list | jq 'group_by(.Node) | map({node: .[0].Node, count: length})'

# Sessions on a specific node that may have failed
curl -s http://127.0.0.1:8500/v1/session/node/<node> | jq length

# Session operation rate (creates, destroys, renews) from telemetry
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep consul.session.apply

# Current leader (writes and session ops go through the leader)
curl -s http://127.0.0.1:8500/v1/status/leader

# Gossip view of membership; look for suspect or failed members
curl -s http://127.0.0.1:8500/v1/agent/members | jq '.[] | {Name, Status, role: .Tags.role}'

# Serf health detail from the local agent
consul info | grep -A 20 serf_lan

# KV keys currently locked to a session
curl -s "http://127.0.0.1:8500/v1/kv/?recurse=true" | jq '.[] | select(.Session != null) | {Key, Session}'

How to diagnose it

1. Confirm the invalidation spike. Pull consul.session.apply from /v1/agent/metrics and look for a burst in session destroy operations. A single node failing produces a vertical step; a renewal-path failure produces a sustained scattered pattern.
2. Map invalidations to nodes. List active sessions by node with /v1/session/list. If one node’s session count dropped to zero while others held steady, suspect node failure or a serfHealth flap on that node. Cross-check with /v1/agent/members and consul info for that node’s gossip state.
3. Classify the trigger. Sessions with a TTL and no node loss: the renewal path failed. Sessions with only serfHealth and no TTL: a gossip flap or check deregistration. Sessions with no TTL and no checks do not invalidate on their own; if those disappear, something is calling /v1/session/destroy explicitly.
4. Correlate with runtime pressure. Short-TTL sessions are fragile under GC pauses, CPU starvation, or network blips. Check consul_runtime_total_gc_pause_ns on the nodes running the locking application, and check client-agent RPC failure counters. A tens-of-milliseconds GC pause is enough to miss a renewal window for a 10-second TTL session.
5. Inspect held keys and behavior. For any KV key tied to a session, confirm whether Behavior is release or delete. This determines whether the key is still present (released) or gone (deleted) after invalidation.
6. Check the lock-delay window. After invalidation, contenders must wait the LockDelay interval before re-acquiring. If leader elections seem delayed, confirm the session’s LockDelay value rather than assuming a hung election.

Metrics and signals to monitor

Fixes

Node failure invalidating all sessions at once

This is expected behavior: Consul cannot keep locks held by a dead node. Confirm the node actually failed (versus a gossip flap) before acting. If the node is genuinely gone, ensure leader election converges quickly and contenders respect the LockDelay window rather than hammering /v1/kv/<key>?acquire=<session> in a tight loop.

Short-TTL sessions the application failed to renew

Any GC pause, CPU starvation, or network blip that exceeds the renewal margin loses the lock. The documented behavior is that sessions may not be reaped for up to twice the configured TTL, so a 10-second TTL can take roughly 20 seconds to actually invalidate. Raise the TTL to give the renewal loop more headroom, renew well before the TTL boundary, and run the renewal loop on a path that is not starved by application work. If you need strict TTL enforcement, note that the consul lock CLI attaches node checks (serfHealth) by default; create sessions manually via the HTTP API excluding node checks when you want TTL-only liveness.

Session leak (created, never destroyed)

Sessions accumulate when applications create sessions without calling /v1/session/destroy. Sessions with a TTL eventually expire; sessions with no TTL and no checks live forever until explicitly destroyed. Identify the source by listing /v1/session/list and correlating Node and Name fields with your services. Fix the application lifecycle so destroy is called on shutdown, and prefer TTL-backed sessions so leaks self-clean.

Gossip flap briefly invalidating health-based sessions

If sessions are tied only to serfHealth, a transient gossip partition can mark the node suspect, flip the check critical, and invalidate every session on the node at once. Avoid relying on serfHealth alone for locks where a brief flap is costly. Either add a TTL as a second liveness signal (so a momentary gossip flap does not instantly cost the lock) or tune gossip suspicion timing on unstable networks. Lock-delay absorbs the immediate re-acquire storm but does not prevent the invalidation itself.

Tuning release vs delete and LockDelay

Choose Behavior deliberately. release keeps the key present with the lock cleared, which lets watchers see a ModifyIndex change and react. delete removes the key entirely, which is cleaner but means consumers that expect the key to exist will see a missing-key error.

LockDelay defaults to 15 seconds. It bounds the thundering herd after invalidation by forcing contenders to wait. The valid range is 0 to 60 seconds. A zero value is intended to disable the delay, but there is a long-standing quirk where a server-side default can override it.

If you need predictable near-zero delay, set a small non-zero value rather than relying on zero to disable. There is no API to release a lock-delay early; all contenders must wait the full interval.

Prevention

• Set TTLs with real renewal margin. Pick a TTL at least several times longer than your worst-case renewal latency, and renew at a fraction of the TTL. Treat the 2x reaping behavior as a floor on how long an invalidated session lingers.
• Add a TTL to health-check-only sessions. serfHealth alone makes locks hostage to gossip timing. A TTL gives the renewal loop a chance to survive a flap.
• Destroy sessions on shutdown. Treat /v1/session/destroy as part of graceful shutdown. The endpoint is idempotent.
• Monitor the session invalidation rate directly. A near-zero steady-state rate with a documented deploy correlation is healthy. Unexplained bursts are the signal to investigate.
• Do not assume sessions are reentrant. Build leader election to expect that a lost lock may already be held by another instance by the time it tries to re-acquire.
• Track session count against expected application count. Divergence in either direction (leaks or drops) is actionable.

How Netdata helps

• Per-second consul.session.apply tracking lets you see the invalidation burst at second resolution rather than minute-aggregated telemetry, which is often what distinguishes a single node failing from a scattered renewal failure.
• Correlating session invalidation with Serf member status (consul_serf_lan_member_status) in the same time window makes it fast to confirm whether a gossip flap or a real node failure triggered the cascade.
• Cross-referencing consul_raft_last_contact and Raft state changes against the invalidation burst shows whether the renewal path was disrupted by Raft instability rather than application-side GC or network blips.
• Go runtime GC pause metrics (consul_runtime_total_gc_pause_ns) on the nodes running the locking application pinpoint renewal-path failures caused by stop-the-world pauses.
• Anomaly detection on session count and invalidation rate surfaces both sudden step drops (node failure) and slow monotonic growth (session leak) without hand-tuned thresholds.

Related guides

• Consul “ACL not found”: requests rejected after a token or policy change
• Consul blocking query accumulation: leaked watches that pile up goroutines
• Consul catalog bloat: too many services and checks slowing everything down
• Consul registration storm: catalog churn overwhelming Raft
• Consul anti-entropy not syncing: local agent state and the catalog drifting apart
• Consul client rpc failed: agents alive but the catalog is going stale
• Consul Connect CA rotation failure: a root roll that never finished
• Consul Connect certificate expired: mTLS handshakes failing across the mesh
• Consul cross-datacenter query failure: prepared-query failover masking a DC outage
• Consul DeregisterCriticalServiceAfter: instances vanishing from the catalog
• Consul DNS latency high: slow lookups stalling connections and failovers
• Consul DNS SERVFAIL: service discovery is broken for your applications