$ guides / consul / consul-deregister-critical-service-after ▌

Operations Guides

Consul DeregisterCriticalServiceAfter: instances vanishing from the catalog

A service that was merely slow to recover has disappeared from the Consul catalog. Healthy-instance count dropped to zero. DNS returns empty results. Load balancers have no targets. Yet the service process is running, the host is up, and the Consul agent on that host is healthy and gossiping. DCSA did exactly what you configured.

DeregisterCriticalServiceAfter (DCSA) is an agent-side reaper that removes a service instance and its checks from the catalog once the check has been critical for a configured duration. It is deliberate garbage collection for stale entries. Leave it unset and critical checks plus their output strings accumulate unbounded in the state store, growing memory and slowing snapshots.

What DCSA is and why it matters

DCSA is a field on a health check, not on the service itself. In HCL or JSON config files, the field is deregister_critical_service_after (snake_case). In the HTTP API JSON payload, it is DeregisterCriticalServiceAfter (PascalCase). Both accept a Go duration string: “10m”, “1h”, “90m”. DCSA was introduced in Consul 0.7.0; Consul 1.0 moved the config-file spelling to snake_case while retaining the old camelCase form as a compatibility alias.

When a service-level check enters critical and stays there past the DCSA duration, the local agent removes the service and its checks from local state. Anti-entropy sync then propagates the removal to the server catalog.

Two timing constraints matter:

Minimum effective timeout is 1 minute. Values below 1m are clamped to 1m. A 5s DCSA does not give you a 5s reaper.
The reaper sweeps every check_reap_interval (default 30 seconds), so actual deregistration can lag the configured duration by up to one sweep interval.

The intent is to reap zombie services whose checks went critical and stayed there. The surprise is that a service merely slow to recover vanishes instead of sitting marked critical. Both behaviors are by design.

How it works

stateDiagram-v2
    [*] --> Registered: agent API register with check
    Registered --> Passing: check succeeds
    Passing --> Critical: check fails
    Critical --> Passing: recovers within DCSA window
    Critical --> Deregistered: critical held past DCSA duration
    Deregistered --> [*]: anti-entropy syncs removal to catalog

The lifecycle:

A service is registered via the Agent API with a service-level check that has DCSA set.
The check runs on the client agent and transitions to critical.
The agent’s reaper notices the check has been critical longer than the DCSA duration.
The agent removes the service and all associated checks from local state.
Anti-entropy sync (default ae_interval 1 minute) propagates the removal to the server catalog.
The instance is gone from discovery. DNS for that service returns one fewer address.

To verify DCSA is set on a live service, check the agent’s view of its checks:

curl -s http://localhost:8500/v1/agent/checks | \
  jq '.[] | select(.ServiceName == "myapp") | {CheckID, Status, DeregisterCriticalServiceAfter}'

If the field is empty or missing, the check has no reaper.

Three constraints bite operators regularly:

DCSA only applies to service-level checks. A check defined outside a service block is node-level. When it goes critical it does not trigger service deregistration. The check must be nested inside the service definition, or associated via ServiceID in the API registration call. This is a frequent cause of “DCSA is not working” reports where the check was defined at node scope.

DCSA only fires for Agent API registrations. Services registered through the Catalog API (PUT /v1/catalog/register) accept the DCSA field in the payload, but no agent runs those checks or evaluates DCSA against them. The check never transitions, never goes critical, and the service sits in the catalog indefinitely. To diagnose: if a service appears in /v1/catalog/service/<name> but not in /v1/agent/services on any node, it was registered via the Catalog API.

DCSA is agent-local. If the agent process dies, the reaper cannot run. Services on that node remain in the catalog as critical until the node is manually deregistered or the agent returns. The serfHealth check that reflects gossip participation is itself a node-level check, so even when it goes critical it does not trigger service DCSA.

The flip side: unset DCSA and catalog accumulation

Leaving DCSA unset is equally dangerous. A critical check with no DCSA never gets reaped. Its check entry, including the output string, sits in the catalog indefinitely. Over time this causes:

State store memory growth proportional to accumulated critical checks and their output text.
Larger Raft snapshots, which take longer to create and restore.
Slower catalog scans for DNS and HTTP API queries, since every stale critical entry is still in the working set.
Stale instances that confuse consumers who expect DCSA cleanup.

Health check result accumulation is unbounded if deregister_critical_service_after is not configured. This is not theoretical in fleets with autoscaling or frequent deploys, where services register and then fail without cleanup.

Where it goes wrong in production

Failure mode	What it looks like	Why it happens
Duration too aggressive	Entire service disappears during a downstream dependency blip; healthy count hits zero	DCSA fires before the dependency recovers, reaping instances that would have come back
Catalog API registration	Check stays critical forever, service never reaped	Catalog API registration bypasses the agent; DCSA field is accepted but no agent evaluates it
Node-level check used by mistake	Service stays registered despite a critical check	Check defined outside the service block does not trigger service deregistration
Connect proxy orphaned	Parent service vanishes but `service-proxy` sidecar lingers	DCSA removes the parent; the proxy’s own check still passes, so the proxy’s DCSA never fires. This version-independent behavior was reported as early as Consul 1.2.2.
Manual catalog deregister bounces back	Service reappears 30 to 60 seconds after you remove it	Agent anti-entropy re-registers the service because the agent still has it locally; use the agent API (`PUT /v1/agent/service/deregister/<service-id>`) on the registering node instead
Node death leaves zombies	Node failed, its services stuck critical indefinitely	Agent is gone, reaper cannot run; serfHealth is node-level and does not trigger service DCSA
TTL check timer stall	TTL check appears passing, DCSA never fires	Goroutine stall in some versions where the TTL timer stops firing after a missed interval; reported against Consul 1.5.2
Client library skips the field	Spring Cloud Consul heartbeat mode leaves DCSA unset	Fixed in spring-cloud-consul v2.1.1.RELEASE; older builds return early from check creation without setting the field

If you depend on DCSA firing for a TTL check that has gone quiet, verify the check is actually still transitioning and not stuck in a stale passing state.

Choosing a duration

There is no universal correct value. The right DCSA depends on how long your worst legitimate dependency outage lasts, how fast your consumers fail over, and how aggressively you want the catalog to self-clean.

Short (1 to 5 minutes): Catalog stays tight, zombies disappear fast, but any dependency blip longer than the window reaps real services. During a thundering herd (a shared downstream dependency failing across many services), a short DCSA converts partial degradation into a full outage by dropping healthy-instance counts to zero across the fleet. First response for that pattern: temporarily increase DCSA to prevent mass deregistration.
Medium (10 to 30 minutes): Absorbs most transient flaps and rolling deploys. Services that are genuinely dead still get reaped within an acceptable window. Reasonable default for most production services.
Long (1 hour or more): Useful for services with slow startup or long dependency recovery, but leans toward the accumulation problem. Critical check output sits in memory for the full window.

A practical approach: set DCSA to roughly 2 to 3 times your expected recovery time for the service’s slowest dependency. If a cache warmup takes 5 minutes, a 15-minute DCSA gives the service room to recover without lingering as a zombie.

For flapping services, note that each transition back to passing resets the DCSA timer. A check that oscillates between passing and critical every 20 seconds will never reach a 1-minute DCSA, so the service stays registered but pollutes healthy-instance counts. DCSA does not solve flapping. It only reaps services that go critical and stay critical.

Interactions with flapping and serfHealth

Check flapping. A check oscillating between passing and critical resets the DCSA clock on every passing transition. The service never gets reaped. Track health check transition rate separately, and handle chronic flappers with explicit maintenance mode (consul maint -service-id=<id> -enable) or manual deregistration rather than relying on DCSA.

serfHealth and node failure. The serfHealth check on every node reflects gossip participation. When a node is marked failed in gossip, serfHealth goes critical. But serfHealth is node-level, so DCSA on it does not deregister the node’s services. Node failure requires either the agent to return and run its reaper, or manual cleanup via the catalog API (PUT /v1/catalog/deregister) or Autopilot dead-server removal. DCSA does not cover the case where the machine running the agent is gone entirely.

If ACLs are enabled, the token used for the propagated Catalog.Deregister RPC must authorize the deletion: node:write on the node or service:write on the affected service. Without an acceptable token, locally-removed services may persist in the server catalog.

Signals to watch

Signal	Why it matters	Warning sign
`consul.catalog.deregister` latency timer (sample rate)	Tracks how fast deregister operations occur	Sudden spike correlating with a dependency outage means DCSA is eating live services
`consul.catalog.register` latency timer (sample rate)	Tracks re-registration churn	Register and deregister rates both high means flapping or anti-entropy fighting itself
Health check critical count	Shows how many checks are sitting critical	Sustained growth with no DCSA set indicates catalog accumulation
Healthy instance count per service	The number consumers actually route to	Drops to zero for a service that is merely degraded, not dead
Health check transition rate	Flapping detection	High transition rate per check means DCSA will never fire for that service
`consul.raft.fsm.apply` latency	Write path sensitivity to catalog churn	Spikes during mass deregistration events indicate the reaper storm is pressuring Raft
Server heap / `runtime.alloc_bytes`	State store size	Steady growth with unset DCSA correlates with critical check accumulation

How Netdata helps

Correlate deregister rate spikes against per-service healthy instance counts in the same view. When a burst lines up with a healthy-count drop to zero, you can see within seconds whether DCSA is the cause rather than a real service failure.
Per-second resolution on health check transitions catches flapping patterns that minute-granularity dashboards miss. A check oscillating every few seconds resets the DCSA timer invisibly at coarser resolution.
Anomaly detection on register and deregister baselines flags reaper storms during dependency outages before healthy-instance counts hit zero, giving you a window to raise DCSA or shed load.
Track server heap and Raft apply latency alongside critical check counts. If heap is growing and DCSA is unset on a class of services, the correlation points at catalog accumulation.
Distinguish agent-local deregistration from catalog-level churn by watching both rates together. A deregister spike with no corresponding register bounce means real removals; matched rates mean anti-entropy disagreement or flapping.