Envoy stuck initializing: waiting on xDS while the pod never turns ready

The pod sits at 0/1 Running indefinitely. kubectl describe shows failing readiness probes, the Envoy sidecar never reports ready, and the application container may be healthy but unreachable because the sidecar gate is closed. Envoy’s /ready admin endpoint returns HTTP 503. /server_info reports PRE_INITIALIZING or INITIALIZING. The process is alive, but it never received its initial xDS configuration, so the listener sockets exist but no listeners are active.

This is not a slow cold start. A healthy Envoy cold start resolves in seconds, not minutes. When the init phase persists, the usual root cause is a slow or unreachable control plane at startup. If a liveness probe also points to /ready with a short timeout, kubelet kills the pod and you get a restart loop that looks like a crash but is a starvation pattern: each restart resets the warming clock and Envoy never gets a fair chance to fetch its first config.

This guide covers distinguishing a healthy cold start from a genuinely stuck initialization, identifying which init phase is blocked, and breaking the liveness-probe feedback loop without masking the underlying control-plane problem.

What this means

Envoy’s init manager runs a multi-phase startup before the proxy considers itself ready. The phases proceed through static and DNS clusters first, then EDS clusters, then CDS (if configured), then LDS and RDS. Listeners do not start accepting connections until every phase completes. The lifecycle states are exposed as a string on /server_info and as the server.state stat: 0=LIVE, 1=DRAINING, 2=PRE_INITIALIZING, 3=INITIALIZING.

The /ready admin endpoint returns HTTP 200 only when state is LIVE. For every other state, including PRE_INITIALIZING, INITIALIZING, and DRAINING, it returns HTTP 503. In Kubernetes, the readiness probe typically hits /ready directly (or, in Istio sidecar mode, port 15021 /healthz/ready). As long as /ready returns 503, the pod stays NotReady and is removed from any EndpointSlices that drive service traffic.

This state is invisible from the data plane. Envoy is not crashing. Memory is stable. There is no error counter climbing. The only signals are server.state being non-zero and the warming counts being non-zero. If you do not actively monitor those signals, the only thing that tells you the pod is stuck is Kubernetes itself: the pod is Running but not Ready.

stateDiagram-v2
    [*] --> PRE_INITIALIZING: process start
    PRE_INITIALIZING --> INITIALIZING: static and DNS clusters resolved
    INITIALIZING --> LIVE: xDS resources fetched and clusters warmed
    LIVE --> DRAINING: SIGTERM or hot restart
    PRE_INITIALIZING --> Stuck: control plane unreachable, timeout 0
    INITIALIZING --> Stuck: cluster warming pending
    Stuck --> PRE_INITIALIZING: liveness probe kills pod, restart loop

The feedback loop to watch for: a stuck init phase makes /ready return 503, which makes the readiness probe fail, which keeps the pod NotReady. If a liveness probe also targets /ready, kubelet restarts the pod. The new pod hits the same control plane, gets stuck again, and the cycle repeats. The crash counter goes up even though Envoy itself never crashed.

Common causes

Quick checks

Run these from inside the pod or against the admin port. In Istio sidecar mode the admin port is 15000 and the health port is 15021. Standalone Envoy typically uses 9901.

# Readiness from Envoy's perspective
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:15000/ready
# 200 = LIVE, 503 = PRE_INITIALIZING / INITIALIZING / DRAINING

# Lifecycle state string
curl -s http://localhost:15000/server_info | jq -r .state

# Numeric state and uptime
curl -s http://localhost:15000/stats | grep -E '^server\.(state|uptime)'

# Control plane connection
curl -s http://localhost:15000/stats | grep control_plane.connected_state

# Warming counts that should be zero in steady state
curl -s http://localhost:15000/stats | grep -E 'warming_clusters|listeners_warming'

# xDS cluster update failures
curl -s http://localhost:15000/stats | grep -E 'update_failure|update_rejected'

# Config that Envoy actually has, with version info
curl -s http://localhost:15000/config_dump | jq '.configs[] | .version_info // empty' | head

From the Kubernetes side, confirm the loop is being driven by probes and not by an actual crash.

# Pod phase, restart count, and container state
kubectl describe pod <pod> | grep -A4 -E 'Containers:|State:|Last State:|Ready:|Restart Count:'

# Readiness and liveness probe definitions
kubectl get pod <pod> -o jsonpath='{.spec.containers[*].readinessProbe}' ; echo
kubectl get pod <pod> -o jsonpath='{.spec.containers[*].livenessProbe}' ; echo

# Recent restart events
kubectl get events --field-selector involvedObject.name=<pod> --sort-by=.lastTimestamp

If Last State: Terminated shows exit code 137 with reason OOMKilled, this is not an init bug; that is a memory kill and a different investigation. If it shows exit code 137 driven by the kubelet killing for liveness, you are in the init-starvation loop.

How to diagnose it

1. Confirm it is initialization, not a hot restart or drain. Check server.state and server.uptime. State 2 or 3 with low uptime points to startup. State 1 (DRAINING) with a sibling process present is a hot restart window and resolves on its own. State 1 sustained without a parent process is an unintended drain and a separate issue.

2. Identify which phase is stuck. /server_info returns PRE_INITIALIZING or INITIALIZING as a string. PRE_INITIALIZING means Envoy has not finished setting up its initial static and DNS clusters, which usually means the xDS server cluster itself is unreachable. INITIALIZING means Envoy is connected or attempting to connect, but is waiting on resources or warming.

3. Check the control plane connection. control_plane.connected_state should be 1 for a pod that has any chance of finishing init. If it is 0, the problem is between Envoy and the control plane: network policy, DNS, the control plane being down, or mTLS bootstrap failure.

4. Check warming counts. cluster_manager.warming_clusters and listener_manager.total_listeners_warming should be zero in steady state. During a healthy startup they are briefly nonzero then drain to zero. If they plateau, a specific cluster is waiting on a dependency: an EDS response that has not arrived, an SDS secret that has not been delivered, or a STRICT_DNS resolution that has not completed.

5. Inspect the xDS cluster directly. Look at the cluster used to reach the control plane (for example xds-grpc or istiod.istio-system.svc). cluster.<xds_cluster>.update_failure and the cluster’s connection stats tell you whether Envoy can talk to the control plane at all. A cluster that resolves DNS but cannot establish the gRPC stream is a different problem from one that cannot resolve at all.

6. Look at config rejection separately. A connected Envoy can still be stuck if every config update is NACKed. cluster.<name>.update_rejected and listener_manager.listener_create_failure indicate the control plane is sending config Envoy considers invalid. This is a different fix path than an unreachable control plane.

7. Read the logs for the specific blocker. Look for “dns resolution without records”, “connect timeout”, gRPC status 14, or the init manager naming an unready target such as a FilterConfigSubscription. The init manager names the specific resource it is waiting on in many builds, which tells you the exact dependency.

Metrics and signals to monitor

Correlate server.state, control_plane.connected_state, warming counts, and server.uptime together. A non-LIVE state with low uptime and a rising restart count is the signature of the init-starvation loop. A non-LIVE state with stable high uptime is a genuine hang during init.

Fixes

The control plane is unreachable at startup

This is the most common cause. Verify the control plane is up and that the pod can reach it: network policy, service resolution, and any admission-injected sidecar bootstrap. If the control plane is overloaded at startup (common in large meshes where every sidecar connects simultaneously), the fix is capacity or staged rollout, not Envoy tuning. Once the control plane is reachable, control_plane.connected_state flips to 1 and init completes without any Envoy restart.

initial_fetch_timeout is set to 0

The Envoy core default for initial_fetch_timeout is 15 seconds; setting it to 0 means Envoy waits indefinitely for the first xDS response. That indefinite wait is desirable in a stable environment, but it is a trap when the control plane is slow at startup: Envoy sits in init forever, the pod never becomes Ready, and nothing fails loudly. Envoy Gateway 1.7.0 changed its default to 0s, so if you upgraded and started seeing pods that never finish init, this is the first place to look. Restore a finite timeout so init fails fast and surfaces the problem instead of hanging.

A liveness probe is driving a crash loop

If the liveness probe targets /ready with a short timeoutSeconds and failureThreshold, kubelet kills the pod before the control plane can deliver the initial config, and the new pod starts the same race. The structural fix is to use a startup probe for the slow init window and keep liveness for steady-state health. A startup probe with a longer periodSeconds and failureThreshold lets Envoy finish warming without being killed. Do not raise the liveness threshold to mask the problem; pair it with investigating why init is slow in the first place. If you only remove the liveness failure, you get pods that stay NotReady forever instead of pods that restart, which is arguably worse because the restart count no longer flags the issue.

A specific cluster is stuck warming

One stuck cluster blocks all listeners, because init does not complete until every cluster has warmed. Use /clusters to find the cluster in warming state, then check what it is waiting for: an EDS response, an SDS secret, a DNS resolution. A known pattern is a static cluster whose eds_service_name collides with a control-plane-generated cluster name, which causes the cluster to wait forever for an EDS response that matches the wrong scope. Remove the collision.

Envoy is connected but rejecting config

If control_plane.connected_state is 1 but update_rejected or listener_create_failure are nonzero, Envoy is receiving config it considers invalid. This is not an init timeout; it is a bad-config push. Look at Envoy’s process logs for the NACK reason, which is logged but not exposed as a stat. Fix the config on the control plane side. Increasing the fetch timeout does not help here because the resources are arriving, they are just being rejected.

Prevention

• Monitor server.state and /ready as first-class signals, with the caveat that non-LIVE during rolling restarts is expected. Alert on non-LIVE combined with low server.uptime past a startup window.
• Alert on control_plane.connected_state = 0 sustained, especially for new pods. This is the leading indicator that init will not complete on its own.
• Use startup probes in Kubernetes, separate from liveness, for any Envoy that depends on a remote control plane. Give init a fair window before liveness can kill the pod.
• Keep initial_fetch_timeout finite in environments where the control plane may be slow at startup. Indefinite waits turn a control-plane problem into an invisible hang.
• Track warming counts in dashboards, not just alerts. A plateau in warming_clusters during steady state is a config convergence problem even if no SLO has tripped yet.
• Keep Envoy current. Warming and init-related bugs have been reported and fixed across versions; running a recent Envoy release reduces exposure to known init-path regressions.

How Netdata helps

• Per-second server.state and /ready status let you see the exact moment a pod enters and exits PRE_INITIALIZING, which is the difference between a healthy cold start and a stuck init.
• Correlating server.state with server.uptime and restart counts surfaces the init-starvation loop directly: non-LIVE state plus repeatedly resetting uptime is a liveness-probe driven crash loop, not an Envoy bug.
• control_plane.connected_state at high resolution shows whether the control plane was reachable at the moment init stalled, which is the single most important fact for routing the investigation.
• Warming counters per cluster and listener let you see if a single cluster is holding up all listeners, or if init is blocked at the connection phase before any warming begins.
• Anomaly detection on startup duration flags pods whose init window is several standard deviations beyond the fleet baseline, so you catch a slow control plane before a liveness probe turns it into a loop.
• xDS update rejection counters distinguish “Envoy never received config” from “Envoy received config and NACKed it”, which have completely different fix paths.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy clusters stuck warming: warming_clusters non-zero and routes returning 503
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy control_plane.connected_state = 0: running on stale xDS config
• Envoy downstream 4xx spike: 401s, 403s, and 404s from the client side
• Envoy downstream connection flood: slowloris, the cx-to-rq ratio, and oversized requests
• Envoy downstream_cx_active growing: connection leaks and idle-timeout gaps
• Envoy downstream_cx_overflow and overload_reject: connections turned away at the door
• Envoy downstream_rq_time high: client-observed latency and proxy overhead