A Kubernetes readiness probe starts failing. The pod shows NotReady, the service stops sending traffic, and a rolling deploy stalls. You exec in and hit Envoy’s /ready endpoint:

# Check readiness state on the admin port
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:9901/ready
503
  1. The probe is doing its job: Envoy is reporting server.state != LIVE. The real question is whether this is expected (rolling deploy, hot restart, normal warm-up) or genuine capacity loss (stuck init, orphaned drain, crashed child). Non-LIVE is normal during deploys, so alerting on server.state != LIVE alone is noisy. You need to combine it with listener state and uptime before paging.

What this means

Envoy exposes its lifecycle as a single gauge, server.state:

ValueState/ready returnsAccepts new connections?
0LIVE200Yes
1DRAINING503No, existing connections drain
2PRE_INITIALIZING503No, process starting, init manager not yet begun
3INITIALIZING503No, init manager running, waiting on initial xDS

You can read the gauge from server.state on the admin stats endpoint, or implicitly via /ready, which returns 503 for any non-LIVE state. The admin port is 9901 in standalone Envoy and 15000 in Istio sidecars. Istio’s health probe lives on port 15021 at /healthz/ready. Do not confuse the two.

A healthy process moves through these states predictably:

stateDiagram-v2
    [*] --> PRE_INITIALIZING: process start
    PRE_INITIALIZING --> INITIALIZING: init manager begins
    INITIALIZING --> LIVE: initial xDS applied, listeners active
    LIVE --> DRAINING: SIGTERM, /drain_listeners, or hot restart parent
    DRAINING --> [*]: parent_shutdown_time_s elapsed

Three facts that catch operators off guard:

  1. Envoy can be LIVE with zero healthy upstreams. server.state = LIVE only confirms the proxy itself is initialized and accepting connections. It says nothing about cluster health. A LIVE proxy with membership_healthy = 0 will return 503 on every request. If the symptom is “Envoy is up but everything 503s”, look at upstream health, not server state.
  2. Non-LIVE is expected during hot restart and rolling deploys. During hot restart, both the old process (DRAINING) and the new process (INITIALIZING to LIVE) run simultaneously. Non-LIVE on one process during a deploy is normal. Page only when the state persists past the drain window or sits in INITIALIZING indefinitely.
  3. The Prometheus metric may lag the admin endpoint in Istio. /stats/prometheus has been reported to keep envoy_server_state at its LIVE value even when the admin logs show DRAINING, particularly in Istio sidecars. Treat the admin endpoint /ready as authoritative, not the scraped metric.

Common causes

CauseWhat it looks likeFirst thing to check
Hot restart in progressTwo Envoy processes running; old process DRAINING; server.parent_connections > 0server.hot_restart_epoch
SIGTERM or rolling deployState transitions LIVE to DRAINING after a termination signal; matches a deploy eventserver.uptime and orchestrator deploy logs
Stuck initializing, xDS unreachablePRE_INITIALIZING or INITIALIZING past 60s with no listener activitycontrol_plane.connected_state
Stuck initializing, dependency missingINITIALIZING with connected_state = 1 but listeners never activatecluster_manager.warming_clusters, listener_manager.total_listeners_warming
Orphaned drainDRAINING sustained past --drain-time-s with no SIGTERM in flight/drain_listeners admin endpoint state, shutdown-manager sidecar
LDS update drained a listenerOne listener drained via LDS push, others still activelistener_manager.total_listeners_active vs total_listeners_warming

Quick checks

# Readiness check (standalone Envoy admin port)
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:9901/ready

# Readiness check (Istio sidecar health port)
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:15021/healthz/ready

# Current state gauge
curl -s http://localhost:9901/stats | grep '^server.state'

# Process metadata: state, uptime, hot restart epoch (JSON, more reliable than scraping)
curl -s http://localhost:9901/server_info | python3 -m json.tool

# Listeners actually bound and accepting traffic
curl -s http://localhost:9901/stats | grep 'listener_manager.total_listeners_active'

# Listeners still warming
curl -s http://localhost:9901/stats | grep 'listener_manager.total_listeners_warming'

# xDS control plane connectivity
curl -s http://localhost:9901/stats | grep 'control_plane.connected_state'

# Draining connections from the parent process (non-zero = hot restart in progress)
curl -s http://localhost:9901/stats | grep 'server.parent_connections'

# Clusters still warming (blocks INITIALIZING to LIVE)
curl -s http://localhost:9901/stats | grep 'cluster_manager.warming_clusters'

# Hot restart epoch (incrementing rapidly = restart loop)
curl -s http://localhost:9901/stats | grep 'server.hot_restart_epoch'

Do not call /drain_listeners as a diagnostic. It is a write operation that begins draining live connections.

How to diagnose it

Work through these in order. The goal is one question: is the proxy truly not serving, or is this a lifecycle transition?

  1. Confirm the state value. Use /server_info (JSON, more reliable than scraping server.state as text). The state field gives the enum name directly. Cross-check with the /ready HTTP code.
  2. Check process uptime. If server.uptime is under 60 seconds, this is cold start. PRE_INITIALIZING and INITIALIZING are expected. Wait.
  3. Identify hot restart versus cold start. If server.hot_restart_epoch is greater than zero and server.parent_connections > 0, the old process is draining into the new one. The old process shows DRAINING; the new process should reach LIVE within seconds. Verify the new process is making progress.
  4. For INITIALIZING or PRE_INITIALIZING stuck past 60s, check xDS. If control_plane.connected_state is 0, Envoy cannot reach the control plane, which blocks the initial config fetch and prevents the INITIALIZING to LIVE transition. The pod will not become ready on its own.
  5. For INITIALIZING with connected_state = 1, check warming clusters and listeners. A cluster stuck in warming (cluster_manager.warming_clusters > 0) blocks listener activation. The usual cause is a missing dependency: an EDS endpoint that never arrives, an SDS secret that never loads, or a DNS resolution that never completes for STRICT_DNS clusters.
  6. For DRAINING sustained past --drain-time-s, suspect an orphaned drain. The default drain time is 600 seconds. The process received SIGTERM (or an admin /drain_listeners call) and never exited. In Kubernetes this is often a shutdown-manager sidecar whose own lifecycle got out of sync, or a long-lived Prometheus scrape connection on the stats listener holding the process open. Check shutdown-manager logs and what is still holding the admin port.
  7. Confirm real capacity loss before paging. Decision rule: server.state != LIVE AND listener_manager.total_listeners_active == 0 AND server.uptime > 300s. All three together is real capacity loss. Any one alone is not.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
server.stateThe state gauge itselfAny non-LIVE value sustained outside a known deploy window
listener_manager.total_listeners_activeListeners actually bound and acceptingDrops to zero on a production instance that should be serving
listener_manager.total_listeners_warmingListeners waiting on dependenciesNon-zero for more than 60 seconds during steady state
cluster_manager.warming_clustersClusters blocking listener activationNon-zero sustained, especially with connected_state = 0
control_plane.connected_statexDS reachability0 for more than 60 seconds during INITIALIZING means stuck startup
server.parent_connectionsOld process draining into newNon-zero means hot restart in progress, expected during deploys
server.hot_restart_epochNumber of hot restartsIncrementing more than once per minute is a restart loop
server.uptimeTime since current process startedLow values combined with non-LIVE means cold start, not failure

Fixes

Stuck INITIALIZING due to control plane unreachable

Verify the control plane independently. For Istio, check Istiod health and pilot_xds_push_errors. For a custom xDS server, check its logs and connectivity. Check network policy and mTLS between Envoy and the control plane. Once control_plane.connected_state returns to 1, Envoy fetches initial config and transitions to LIVE on its own. Do not restart Envoy first; if the control plane is the problem, restarting just resets the timer.

Stuck INITIALIZING due to a cluster that never warms

Identify which cluster is blocking. cluster_manager.warming_clusters is non-zero. Use /config_dump?include_eds to see which clusters have no endpoints. For STRICT_DNS clusters, check cluster.<name>.update_failure. For EDS clusters, the control plane is not sending endpoints. For SDS dependencies, check secret resolution. Removing the stuck cluster from config (if non-essential) unblocks the rest of the listeners.

Orphaned DRAINING process

If a deploy left an Envoy in DRAINING that never exited, the shutdown-manager sidecar pattern is the likely culprit. The immediate fix is to terminate the process. The durable fix is to ensure the shutdown-manager has a working liveness probe (or remove a spurious one) and that long-lived stats scrape connections cannot hold the drain open. In Envoy Gateway, reducing shutdown.drainTimeout to 60 seconds is a common mitigation.

Hot restart race

If both processes are non-LIVE simultaneously during a hot restart, FD budget may be the constraint: both processes hold FDs during the handoff. If baseline FD usage is above 50%, hot restart can briefly exhaust FDs and drop connections. Either raise the FD limit or schedule hot restarts during lower traffic. Hot restart is not supported on Windows. A concurrency decrease between epochs can also drop connections.

Prevention

  • Alert on the combination, not the state alone. Page on server.state != LIVE AND listener_manager.total_listeners_active == 0 AND uptime > 300s. Alert separately on INITIALIZING sustained past the configured initial_fetch_timeout for cold starts.
  • Bound the cold start window. Configure initial_fetch_timeout explicitly rather than relying on the default. If Envoy cannot fetch initial config within the bound, it should fail rather than hang in INITIALIZING forever.
  • Make readiness probes tolerant of warm-up. A readiness probe with initialDelaySeconds too low will kill Envoy during a slow control-plane start and create a crash loop. Set the initial delay to at least 60 seconds and use failureThreshold greater than 1.
  • Monitor hot restart epoch rate. server.hot_restart_epoch incrementing more than once per minute indicates a restart loop, not a planned deploy.
  • Watch FD budget before deploys. Hot restart briefly doubles FD usage. Keep baseline under 50% of the limit so the rollover has headroom.
  • Treat the admin endpoint as authoritative in Istio. Because the scraped envoy_server_state metric may lag the admin endpoint, page on the /healthz/ready HTTP code, not on the metric alone.

How Netdata helps

  • Per-second server.state resolution shows the exact moment a state transition happens and correlates it with deploy events, xDS pushes, or SIGTERM signals without inferring the timeline from 30-second Prometheus scrapes.
  • Correlate server.state with listener_manager.total_listeners_active and cluster_manager.warming_clusters in a single view to distinguish a stuck startup from a healthy warm-up.
  • ML-based anomaly detection on server.hot_restart_epoch rate catches restart loops that look legitimate to a fixed threshold, since each individual restart is normal.
  • Composite alerts page only on the playbook combination: non-LIVE state AND zero active listeners AND uptime past 300 seconds, suppressing noise from routine deploys.
  • Sidecar-aware collection surfaces Istio’s 15021 health endpoint alongside the 9901 admin port, so the two signals read together regardless of deployment variant.