A service that was merely slow to recover has disappeared from the Consul catalog. Healthy-instance count dropped to zero. DNS returns empty results. Load balancers have no targets. Yet the service process is running, the host is up, and the Consul agent on that host is healthy and gossiping. DCSA did exactly what you configured.

DeregisterCriticalServiceAfter (DCSA) is an agent-side reaper that removes a service instance and its checks from the catalog once the check has been critical for a configured duration. It is deliberate garbage collection for stale entries. Leave it unset and critical checks plus their output strings accumulate unbounded in the state store, growing memory and slowing snapshots.

What DCSA is and why it matters

DCSA is a field on a health check, not on the service itself. In HCL or JSON config files, the field is deregister_critical_service_after (snake_case). In the HTTP API JSON payload, it is DeregisterCriticalServiceAfter (PascalCase). Both accept a Go duration string: “10m”, “1h”, “90m”.

When a service-level check enters critical and stays there past the DCSA duration, the local agent removes the service and its checks from local state. Anti-entropy sync then propagates the removal to the server catalog.

Two timing constraints matter:

  • Minimum effective timeout is 1 minute. Values below 1m are clamped to 1m. A 5s DCSA does not give you a 5s reaper.
  • The reaper sweeps periodically, so actual deregistration can lag the configured duration by up to one sweep interval.

The intent is to reap zombie services whose checks went critical and stayed there. The surprise is that a service merely slow to recover vanishes instead of sitting marked critical. Both behaviors are by design.

How it works

stateDiagram-v2
    [*] --> Registered: agent API register with check
    Registered --> Passing: check succeeds
    Passing --> Critical: check fails
    Critical --> Passing: recovers within DCSA window
    Critical --> Deregistered: critical held past DCSA duration
    Deregistered --> [*]: anti-entropy syncs removal to catalog

The lifecycle:

  1. A service is registered via the Agent API with a service-level check that has DCSA set.
  2. The check runs on the client agent and transitions to critical.
  3. The agent’s reaper notices the check has been critical longer than the DCSA duration.
  4. The agent removes the service and all associated checks from local state.
  5. Anti-entropy sync propagates the removal to the server catalog.
  6. The instance is gone from discovery. DNS for that service returns one fewer address.

To verify DCSA is set on a live service, check the agent’s view of its checks:

curl -s http://localhost:8500/v1/agent/checks | \
  jq '.[] | select(.ServiceName == "myapp") | {CheckID, Status, DeregisterCriticalServiceAfter}'

If the field is empty or missing, the check has no reaper.

Three constraints bite operators regularly:

DCSA only applies to service-level checks. A check defined outside a service block is node-level. When it goes critical it does not trigger service deregistration. The check must be nested inside the service definition, or associated via ServiceID in the API registration call. This is a frequent cause of “DCSA is not working” reports where the check was defined at node scope.

DCSA only fires for Agent API registrations. Services registered through the Catalog API (PUT /v1/catalog/register) accept the DCSA field in the payload, but no agent runs those checks or evaluates DCSA against them. The check never transitions, never goes critical, and the service sits in the catalog indefinitely. To diagnose: if a service appears in /v1/catalog/service/<name> but not in /v1/agent/services on any node, it was registered via the Catalog API.

DCSA is agent-local. If the agent process dies, the reaper cannot run. Services on that node remain in the catalog as critical until the node is manually deregistered or the agent returns. The serfHealth check that reflects gossip participation is itself a node-level check, so even when it goes critical it does not trigger service DCSA.

The flip side: unset DCSA and catalog accumulation

Leaving DCSA unset is equally dangerous. A critical check with no DCSA never gets reaped. Its check entry, including the output string, sits in the catalog indefinitely. Over time this causes:

  • State store memory growth proportional to accumulated critical checks and their output text.
  • Larger Raft snapshots, which take longer to create and restore.
  • Slower catalog scans for DNS and HTTP API queries, since every stale critical entry is still in the working set.
  • Stale instances that confuse consumers who expect DCSA cleanup.

Health check result accumulation is unbounded if deregister_critical_service_after is not configured. This is not theoretical in fleets with autoscaling or frequent deploys, where services register and then fail without cleanup.

Where it goes wrong in production

Failure modeWhat it looks likeWhy it happens
Duration too aggressiveEntire service disappears during a downstream dependency blip; healthy count hits zeroDCSA fires before the dependency recovers, reaping instances that would have come back
Catalog API registrationCheck stays critical forever, service never reapedCatalog API registration bypasses the agent; DCSA field is accepted but no agent evaluates it
Node-level check used by mistakeService stays registered despite a critical checkCheck defined outside the service block does not trigger service deregistration
Connect proxy orphanedParent service vanishes but service-proxy sidecar lingersDCSA removes the parent; the proxy’s own check still passes, so the proxy’s DCSA never fires.
Manual catalog deregister bounces backService reappears 30 to 60 seconds after you remove itAgent anti-entropy re-registers the service because the agent still has it locally; use the agent API (PUT /v1/agent/service/deregister/<service-id>) on the registering node instead
Node death leaves zombiesNode failed, its services stuck critical indefinitelyAgent is gone, reaper cannot run; serfHealth is node-level and does not trigger service DCSA
TTL check timer stallTTL check appears passing, DCSA never firesGoroutine stall in some versions where the TTL timer stops firing after a missed interval
Client library skips the fieldSpring Cloud Consul heartbeat mode leaves DCSA unsetOlder builds return early from check creation without setting the field

If you depend on DCSA firing for a TTL check that has gone quiet, verify the check is actually still transitioning and not stuck in a stale passing state.

Choosing a duration

There is no universal correct value. The right DCSA depends on how long your worst legitimate dependency outage lasts, how fast your consumers fail over, and how aggressively you want the catalog to self-clean.

  • Short (1 to 5 minutes): Catalog stays tight, zombies disappear fast, but any dependency blip longer than the window reaps real services. During a thundering herd (a shared downstream dependency failing across many services), a short DCSA converts partial degradation into a full outage by dropping healthy-instance counts to zero across the fleet. First response for that pattern: temporarily increase DCSA to prevent mass deregistration.
  • Medium (10 to 30 minutes): Absorbs most transient flaps and rolling deploys. Services that are genuinely dead still get reaped within an acceptable window. Reasonable default for most production services.
  • Long (1 hour or more): Useful for services with slow startup or long dependency recovery, but leans toward the accumulation problem. Critical check output sits in memory for the full window.

A practical approach: set DCSA to roughly 2 to 3 times your expected recovery time for the service’s slowest dependency. If a cache warmup takes 5 minutes, a 15-minute DCSA gives the service room to recover without lingering as a zombie.

For flapping services, note that each transition back to passing resets the DCSA timer. A check that oscillates between passing and critical every 20 seconds will never reach a 1-minute DCSA, so the service stays registered but pollutes healthy-instance counts. DCSA does not solve flapping. It only reaps services that go critical and stay critical.

Interactions with flapping and serfHealth

Check flapping. A check oscillating between passing and critical resets the DCSA clock on every passing transition. The service never gets reaped. Track health check transition rate separately, and handle chronic flappers with explicit maintenance mode (consul maint -service-id=<id> -enable) or manual deregistration rather than relying on DCSA.

serfHealth and node failure. The serfHealth check on every node reflects gossip participation. When a node is marked failed in gossip, serfHealth goes critical. But serfHealth is node-level, so DCSA on it does not deregister the node’s services. Node failure requires either the agent to return and run its reaper, or manual cleanup via the catalog API (PUT /v1/catalog/deregister) or Autopilot dead-server removal. DCSA does not cover the case where the machine running the agent is gone entirely.

If ACLs are enabled, the agent needs a token with service:write for the affected services to propagate deregistrations via anti-entropy. Without it, locally-removed services may persist in the server catalog.

Signals to watch

SignalWhy it mattersWarning sign
consul.catalog.deregister rateTracks how fast services are being reapedSudden spike correlating with a dependency outage means DCSA is eating live services
consul.catalog.register rateTracks re-registration churnRegister and deregister rates both high means flapping or anti-entropy fighting itself
Health check critical countShows how many checks are sitting criticalSustained growth with no DCSA set indicates catalog accumulation
Healthy instance count per serviceThe number consumers actually route toDrops to zero for a service that is merely degraded, not dead
Health check transition rateFlapping detectionHigh transition rate per check means DCSA will never fire for that service
consul.raft.fsm.apply latencyWrite path sensitivity to catalog churnSpikes during mass deregistration events indicate the reaper storm is pressuring Raft
Server heap / runtime.alloc_bytesState store sizeSteady growth with unset DCSA correlates with critical check accumulation

How Netdata helps

  • Correlate deregister rate spikes against per-service healthy instance counts in the same view. When a burst lines up with a healthy-count drop to zero, you can see within seconds whether DCSA is the cause rather than a real service failure.
  • Per-second resolution on health check transitions catches flapping patterns that minute-granularity dashboards miss. A check oscillating every few seconds resets the DCSA timer invisibly at coarser resolution.
  • Anomaly detection on register and deregister baselines flags reaper storms during dependency outages before healthy-instance counts hit zero, giving you a window to raise DCSA or shed load.
  • Track server heap and Raft apply latency alongside critical check counts. If heap is growing and DCSA is unset on a class of services, the correlation points at catalog accumulation.
  • Distinguish agent-local deregistration from catalog-level churn by watching both rates together. A deregister spike with no corresponding register bounce means real removals; matched rates mean anti-entropy disagreement or flapping.