$ guides / envoy / envoy-server-state-not-live ▌

Operations Guides

Envoy server.state not LIVE: draining, initializing, and the /ready probe

A Kubernetes readiness probe starts failing. The pod shows NotReady, the service stops sending traffic, and a rolling deploy stalls. You exec in and hit Envoy’s /ready endpoint:

# Check readiness state on the admin port
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:9901/ready
503

The probe is doing its job: Envoy is reporting server.state != LIVE. The real question is whether this is expected (rolling deploy, hot restart, normal warm-up) or genuine capacity loss (stuck init, orphaned drain, crashed child). Non-LIVE is normal during deploys, so alerting on server.state != LIVE alone is noisy. You need to combine it with listener state and uptime before paging.

What this means

Envoy exposes its lifecycle as a single gauge, server.state:

Value	State	`/ready` returns	Accepts new connections?
0	LIVE	200	Yes
1	DRAINING	503	No, existing connections drain
2	PRE_INITIALIZING	503	No, process starting, init manager not yet begun
3	INITIALIZING	503	No, init manager running, waiting on initial xDS

You can read the gauge from server.state on the admin stats endpoint, or implicitly via /ready, which returns 503 for any non-LIVE state. The admin port is 9901 in standalone Envoy and 15000 in Istio sidecars. Istio’s health probe lives on port 15021 at /healthz/ready. Do not confuse the two.

A healthy process moves through these states predictably:

stateDiagram-v2
    [*] --> PRE_INITIALIZING: process start
    PRE_INITIALIZING --> INITIALIZING: init manager begins
    INITIALIZING --> LIVE: initial xDS applied, listeners active
    LIVE --> DRAINING: SIGTERM, /drain_listeners, or hot restart parent
    DRAINING --> [*]: parent_shutdown_time_s elapsed

Three facts that catch operators off guard:

Envoy can be LIVE with zero healthy upstreams. server.state = LIVE only confirms the proxy itself is initialized and accepting connections. It says nothing about cluster health. A LIVE proxy with membership_healthy = 0 will return 503 on every request. If the symptom is “Envoy is up but everything 503s”, look at upstream health, not server state.
Non-LIVE is expected during hot restart and rolling deploys. During hot restart, both the old process (DRAINING) and the new process (INITIALIZING to LIVE) run simultaneously. Non-LIVE on one process during a deploy is normal. Page only when the state persists past the drain window or sits in INITIALIZING indefinitely.
The Prometheus metric may lag the admin endpoint in Istio. /stats/prometheus has been reported to keep envoy_server_state at its LIVE value even when the admin logs show DRAINING, particularly in Istio sidecars. Treat the admin endpoint /ready as authoritative, not the scraped metric.

Common causes

Cause	What it looks like	First thing to check
Hot restart in progress	Two Envoy processes running; old process DRAINING; `server.parent_connections > 0`	`server.hot_restart_epoch`
SIGTERM or rolling deploy	State transitions LIVE to DRAINING after a termination signal; matches a deploy event	`server.uptime` and orchestrator deploy logs
Stuck initializing, xDS unreachable	PRE_INITIALIZING or INITIALIZING past 60s with no listener activity	`control_plane.connected_state`
Stuck initializing, dependency missing	INITIALIZING with `connected_state = 1` but listeners never activate	`cluster_manager.warming_clusters`, `listener_manager.total_listeners_warming`
Orphaned drain	DRAINING sustained past `--drain-time-s` with no SIGTERM in flight	`/drain_listeners` admin endpoint state, shutdown-manager sidecar
LDS update drained a listener	One listener drained via LDS push, others still active	`listener_manager.total_listeners_active` vs `total_listeners_warming`

Quick checks

# Readiness check (standalone Envoy admin port)
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:9901/ready

# Readiness check (Istio sidecar health port)
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:15021/healthz/ready

# Current state gauge
curl -s http://localhost:9901/stats | grep '^server.state'

# Process metadata: state, uptime, hot restart epoch (JSON, more reliable than scraping)
curl -s http://localhost:9901/server_info | python3 -m json.tool

# Listeners actually bound and accepting traffic
curl -s http://localhost:9901/stats | grep 'listener_manager.total_listeners_active'

# Listeners still warming
curl -s http://localhost:9901/stats | grep 'listener_manager.total_listeners_warming'

# xDS control plane connectivity
curl -s http://localhost:9901/stats | grep 'control_plane.connected_state'

# Draining connections from the parent process (non-zero = hot restart in progress)
curl -s http://localhost:9901/stats | grep 'server.parent_connections'

# Clusters still warming (blocks INITIALIZING to LIVE)
curl -s http://localhost:9901/stats | grep 'cluster_manager.warming_clusters'

# Hot restart epoch (incrementing rapidly = restart loop)
curl -s http://localhost:9901/stats | grep 'server.hot_restart_epoch'

Do not call /drain_listeners as a diagnostic. It is a write operation that begins draining live connections.

How to diagnose it

Work through these in order. The goal is one question: is the proxy truly not serving, or is this a lifecycle transition?

Confirm the state value. Use /server_info (JSON, more reliable than scraping server.state as text). The state field gives the enum name directly. Cross-check with the /ready HTTP code.
Check process uptime. If server.uptime is under 60 seconds, this is cold start. PRE_INITIALIZING and INITIALIZING are expected. Wait.
Identify hot restart versus cold start. If server.hot_restart_epoch is greater than zero and server.parent_connections > 0, the old process is draining into the new one. The old process shows DRAINING; the new process should reach LIVE within seconds. Verify the new process is making progress.
For INITIALIZING or PRE_INITIALIZING stuck past 60s, check xDS. If control_plane.connected_state is 0, Envoy cannot reach the control plane, which blocks the initial config fetch and prevents the INITIALIZING to LIVE transition. The pod will not become ready on its own.
For INITIALIZING with connected_state = 1, check warming clusters and listeners. A cluster stuck in warming (cluster_manager.warming_clusters > 0) blocks listener activation. The usual cause is a missing dependency: an EDS endpoint that never arrives, an SDS secret that never loads, or a DNS resolution that never completes for STRICT_DNS clusters.
For DRAINING sustained past --drain-time-s, suspect an orphaned drain. The default drain time is 600 seconds. The process received SIGTERM (or an admin /drain_listeners call) and never exited. In Kubernetes this is often a shutdown-manager sidecar whose own lifecycle got out of sync, or a long-lived Prometheus scrape connection on the stats listener holding the process open. Check shutdown-manager logs and what is still holding the admin port.
Confirm real capacity loss before paging. Decision rule: server.state != LIVE AND listener_manager.total_listeners_active == 0 AND server.uptime > 300s. All three together is real capacity loss. Any one alone is not.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`server.state`	The state gauge itself	Any non-LIVE value sustained outside a known deploy window
`listener_manager.total_listeners_active`	Listeners actually bound and accepting	Drops to zero on a production instance that should be serving
`listener_manager.total_listeners_warming`	Listeners waiting on dependencies	Non-zero for more than 60 seconds during steady state
`cluster_manager.warming_clusters`	Clusters blocking listener activation	Non-zero sustained, especially with `connected_state = 0`
`control_plane.connected_state`	xDS reachability	0 for more than 60 seconds during INITIALIZING means stuck startup
`server.parent_connections`	Old process draining into new	Non-zero means hot restart in progress, expected during deploys
`server.hot_restart_epoch`	Number of hot restarts	Incrementing more than once per minute is a restart loop
`server.uptime`	Time since current process started	Low values combined with non-LIVE means cold start, not failure

Fixes

Stuck INITIALIZING due to control plane unreachable

Verify the control plane independently. For Istio, check Istiod health and pilot_xds_push_errors. For a custom xDS server, check its logs and connectivity. Check network policy and mTLS between Envoy and the control plane. Once control_plane.connected_state returns to 1, Envoy fetches initial config and transitions to LIVE on its own. Do not restart Envoy first; if the control plane is the problem, restarting just resets the timer.

Stuck INITIALIZING due to a cluster that never warms

Identify which cluster is blocking. cluster_manager.warming_clusters is non-zero. Use /config_dump?include_eds to see which clusters have no endpoints. For STRICT_DNS clusters, check cluster.<name>.update_failure. For EDS clusters, the control plane is not sending endpoints. For SDS dependencies, check secret resolution. Removing the stuck cluster from config (if non-essential) unblocks the rest of the listeners.

Orphaned DRAINING process

If a deploy left an Envoy in DRAINING that never exited, the shutdown-manager sidecar pattern is the likely culprit. The immediate fix is to terminate the process. The durable fix is to ensure the shutdown-manager has a working liveness probe (or remove a spurious one) and that long-lived stats scrape connections cannot hold the drain open. In Envoy Gateway, reducing shutdown.drainTimeout to 60 seconds is a common mitigation.

Hot restart race

If both processes are non-LIVE simultaneously during a hot restart, FD budget may be the constraint: both processes hold FDs during the handoff. If baseline FD usage is above 50%, hot restart can briefly exhaust FDs and drop connections. Either raise the FD limit or schedule hot restarts during lower traffic. Hot restart is not supported on Windows. A concurrency decrease between epochs can also drop connections.

Prevention

Alert on the combination, not the state alone. Page on server.state != LIVE AND listener_manager.total_listeners_active == 0 AND uptime > 300s. Alert separately on INITIALIZING sustained past the configured initial_fetch_timeout for cold starts.
Bound the cold start window. Configure initial_fetch_timeout explicitly rather than relying on the default. If Envoy cannot fetch initial config within the bound, it should fail rather than hang in INITIALIZING forever.
Make readiness probes tolerant of warm-up. A readiness probe with initialDelaySeconds too low will kill Envoy during a slow control-plane start and create a crash loop. Set the initial delay to at least 60 seconds and use failureThreshold greater than 1.
Monitor hot restart epoch rate. server.hot_restart_epoch incrementing more than once per minute indicates a restart loop, not a planned deploy.
Watch FD budget before deploys. Hot restart briefly doubles FD usage. Keep baseline under 50% of the limit so the rollover has headroom.
Treat the admin endpoint as authoritative in Istio. Because the scraped envoy_server_state metric may lag the admin endpoint, page on the /healthz/ready HTTP code, not on the metric alone.

How Netdata helps

Per-second server.state resolution shows the exact moment a state transition happens and correlates it with deploy events, xDS pushes, or SIGTERM signals without inferring the timeline from 30-second Prometheus scrapes.
Correlate server.state with listener_manager.total_listeners_active and cluster_manager.warming_clusters in a single view to distinguish a stuck startup from a healthy warm-up.
ML-based anomaly detection on server.hot_restart_epoch rate catches restart loops that look legitimate to a fixed threshold, since each individual restart is normal.
Composite alerts page only on the playbook combination: non-LIVE state AND zero active listeners AND uptime past 300 seconds, suppressing noise from routine deploys.
Sidecar-aware collection surfaces Istio’s 15021 health endpoint alongside the 9901 admin port, so the two signals read together regardless of deployment variant.