Consul service has zero healthy instances: discovery returns nothing

A consumer calls /v1/health/service/<name>?passing=true and gets []. DNS lookups for <name>.service.consul return nothing. Every downstream consumer treats the service as gone. This is total unavailability for that one service, even if the rest of the Consul cluster is healthy.

Two consumer surfaces are affected at once. API consumers see the empty array directly. DNS consumers (port 8600) get a negative response, and clients may cache it. Recovery from zero means both fixing the checks and flushing the negative caches downstream.

There are two distinct flavors of this incident and the fix paths diverge sharply. Either the instances are still registered in the catalog but every check is critical, or the instances have been deregistered entirely (often by DeregisterCriticalServiceAfter) and no longer appear in /v1/catalog/service/<name>. Settle which one you are in first: re-registering instances is a different operation from clearing a check.

There is also a runway you probably missed. Zero is the bottom of a slope. A service dropping from N instances to one or two is the “traffic is about to overwhelm the survivors” signal. Alerting only on zero catches the cliff, not the approach.

What this means

?passing=true is a strict filter. An instance appears in the result only when every check attached to it is passing. That includes the auto-generated serfHealth check on the node. If serfHealth is critical because the agent went down, every service on that node disappears from ?passing=true results even if the service-level checks were fine.

Discovery has two surfaces that share the same catalog but read it differently. The HTTP API gives you control over consistency mode (stale, default, consistent). DNS uses stale consistency by default, so a freshly corrected catalog may still serve stale DNS responses, and downstream resolvers (dnsmasq, systemd-resolved, the OS) may cache them further. A “fixed in Consul but still failing in the app” gap is almost always DNS caching.

Newly registered checks start in the critical state by default. A freshly registered instance will not appear in ?passing=true results until the first successful check execution lands. On rolling restarts or mass re-registrations, this creates a brief zero-healthy window that is normal, not a bug.

flowchart TD
  A["?passing=true returns empty"] --> B["Query /v1/catalog/service/"]
  B --> C{"Instances in catalog?"}
  C -->|Yes| D["Checks critical, instances still registered"]
  C -->|No| E["Instances deregistered"]
  D --> F["Read /v1/health/state/critical for Output"]
  D --> G["Check serfHealth per node"]
  E --> H["Check DeregisterCriticalServiceAfter timer"]
  E --> I["Verify agents hosting instances are alive"]
  F --> J["Treat as service or check failure"]
  G --> K["Treat as node or agent failure"]
  H --> L["Treat as registration lifecycle issue"]

Common causes

Quick checks

Run these read-only commands first. They distinguish “checks critical” from “instances deregistered” and tell you whether you are fighting a real outage or a stale-read illusion.

# Is it really zero? Compare healthy vs total.
SVC=myservice
echo "passing:  $(curl -s "http://127.0.0.1:8500/v1/health/service/$SVC?passing=true" | jq 'length')"
echo "any:      $(curl -s "http://127.0.0.1:8500/v1/health/service/$SVC" | jq 'length')"
echo "catalog:  $(curl -s "http://127.0.0.1:8500/v1/catalog/service/$SVC" | jq 'length')"

# What do the check Output fields say? This is the diagnostic text.
curl -s "http://127.0.0.1:8500/v1/health/service/$SVC" \
  | jq '.[].Checks[] | {CheckID, Status, Output}'

# serfHealth per owning node. If these are critical, the problem is the
# node or agent, not the service check.
curl -s "http://127.0.0.1:8500/v1/health/service/$SVC" \
  | jq -r '.[] | "\(.Node): serfHealth=\(.Checks[] | select(.CheckID=="serfHealth") | .Status)"'

# Is the disagreement a stale-read artifact? Query every server in consistent mode.
for s in server1 server2 server3; do
  printf "%s: " "$s"
  curl -s "http://$s:8500/v1/health/service/$SVC?passing=true&consistent=true" | jq 'length'
done

# DNS path, direct, bypassing downstream resolvers.
dig @127.0.0.1 -p 8600 "$SVC.service.consul" SRV

# Cluster still has a leader? Writes (re-registration) need this.
curl -s http://127.0.0.1:8500/v1/status/leader

# Any registration/deregister churn visible in metrics?
curl -s "http://127.0.0.1:8500/v1/agent/metrics?format=prometheus" \
  | grep -Ei 'catalog.*register'

If passing is zero but catalog is non-zero, instances exist but all checks are critical. If catalog is also zero or smaller than the expected fleet, instances have been deregistered and the issue is registration lifecycle, not check health.

How to diagnose it

1. Settle the “critical vs deregistered” question first. Compare /v1/health/service/$SVC, /v1/health/service/$SVC?passing=true, and /v1/catalog/service/$SVC counts as shown above. The shape of the gap determines which branch you are in.
2. Read the Output field on every critical check. The status tells you nothing actionable; the output tells you whether you have a refused connection, a timeout, an expired certificate, or a check script that exited non-zero. This is the single highest-signal field in the incident.
3. Cross-reference with the owning nodes’ serfHealth. If serfHealth is critical on the nodes hosting the service, the root cause is node or agent health, not the service. Fixing the check is wasted effort until the agent is back.
4. Look for a deploy-shaped cliff. If all instances went critical within the same check interval of a deploy, the new version is failing the check. The check is doing its job.
5. Look for an AZ-shaped cliff. If the affected nodes cluster in one AZ or subnet, suspect an infrastructure event. Cross-reference host-level signals.
6. Rule out stale reads. Query each server directly in consistent mode. If they disagree, you have a catalog consistency problem, not a service health problem. Stale reads can show more healthy instances than actually exist, which is the dangerous direction: clients route to dead endpoints.
7. Check the DeregisterCriticalServiceAfter value on the service registration. With a short value , a check that stays critical for a few minutes can silently deregister the instance. There is no automatic re-registration; you have to put it back.
8. Verify the DNS path separately. Even after the catalog is correct, DNS clients may be holding negative responses. Confirm with a direct dig against the agent before chasing application reports.

Metrics and signals to monitor

Fixes

Pick the branch by diagnosis, not by habit. Restarting the agent is rarely the right first move and can make things worse by re-triggering the critical-on-startup window.

If checks are critical but instances are still registered

The instances are visible to Consul. The fix is on the service or its dependencies.

• Read Output on every critical check. Refused connection usually means the service process is down. Timeout usually means the service is saturated or stuck. TLS errors usually mean certificate expiry or CA mismatch.
• If a deploy shipped a version that fails its check, roll back. Do not loosen the check to make the deploy green; that hides the regression and trains the team to ignore the signal.
• If host resources are exhausted (CPU pinned, fd limit hit, memory pressure), the fix is on the host, not on Consul. The check is correctly reporting that the service cannot do its job.
• If the check criteria themselves are wrong after a config change, fix the check definition. Bring the new criteria to the service owner first; silent check loosening is how services stop being monitored in practice.

If instances have been deregistered

The catalog no longer has them. Recovery requires re-registration, and whatever caused the deregistration must be fixed first or the new registration will deregister again.

• Confirm DeregisterCriticalServiceAfter is the mechanism. If your fleet relies on the agent’s local service definitions, restarting the agent (or sending it a SIGHUP that re-reads its config) will re-register from local state. If registrations were done via the HTTP API, you have to re-register explicitly.
• Verify the underlying check now passes before re-registering, otherwise you are starting the deregister countdown again.
• If you want a wider safety margin during incidents, raise DeregisterCriticalServiceAfter. Short values make outages quieter and recoveries more manual. Long values keep instances around but let dead entries accumulate. Pick per service, not globally.

If the disagreement is a stale-read problem

This is the most dangerous variant because some clients see healthy instances that are actually dead.

• Query each server directly with consistent=true. If they disagree, you have a catalog consistency problem on the server side, not a discovery problem on the client side.
• After the catalog is consistent, flush downstream DNS caches. Negative responses can persist for the resolver’s negative TTL, which on some operating systems defaults to several minutes.
• For application clients using the HTTP API, ensure they are using ?passing=true and an appropriate consistency mode. Without ?passing=true, the API returns all instances including critical ones, which can route traffic to dead endpoints.

Prevention

• Alert on the runway, not the cliff. Per-service thresholds on passing instance count, with the threshold above the minimum needed for survival, gives you lead time. Alerting only on zero gives you none.
• Treat DeregisterCriticalServiceAfter as a deliberate trade-off. Short values keep the catalog clean during real crashes but turn every check storm into a mass deregistration. Long values keep instances around but let dead entries accumulate. Pick per service, not globally.
• Validate checks against real failures periodically. A check that never goes critical during a real outage is worse than no check, because it suppresses the signal. Inject a failure and measure detection latency.
• Track the gap between /v1/health/service/$SVC?passing=true and /v1/catalog/service/$SVC over time. A growing gap means checks are staying critical long enough to be the steady state, which is usually a config or capacity issue.
• For DNS consumers, document the negative-caching behavior. Knowing the cache TTL up front saves hours of “we fixed it but the app is still failing” debugging.

How Netdata helps

• Per-second per-service health instance counts let you see the runway to zero before you hit it, and see recovery the moment it starts, instead of waiting for a poll interval.
• Correlating passing/critical instance counts with consul.client.rpc.failed on the owning agents tells you immediately whether zero is a service problem or an agent-to-server pipeline problem.
• Correlating instance count drops with consul.catalog.deregister rates distinguishes “checks critical” from “instances gone” without manual API calls.
• ML anomaly detection on per-service check transition rates surfaces the flapping-before-the-storm pattern, where checks oscillate and then collapse, earlier than static thresholds.
• DNS stale query metrics alongside catalog metrics show the gap between “fixed in Consul” and “fixed for clients.”
• Host-level CPU, memory, file descriptor, and disk signals on the same timeline as Consul metrics let you confirm or rule out host resource exhaustion without switching tools.

Related guides

• Consul client rpc failed: agents alive but the catalog is going stale
• Consul on EBS: burst-credit exhaustion and the sudden latency cliff
• Consul gossip encryption key mismatch: a botched keyring rotation splits the pool
• Consul gossip flapping: nodes oscillating between alive, suspect, and failed
• Consul serf queue backlog: an agent falling behind on gossip
• Consul gossip storm after mass recovery: rejoin floods and anti-entropy spikes
• How Consul actually works in production: a mental model for operators
• Consul leader election storm: repeated elections and rolling write outages
• Consul monitoring checklist: the signals every production cluster needs
• Consul monitoring maturity model: from survival to expert
• Consul “No cluster leader”: every write is failing
• Consul raft commitTime high: the write pipeline is slowing down