Envoy hot restart races: draining, epoch churn, and dropped connections

You deployed a new Envoy binary or triggered a reload that invoked a hot restart. Seconds later, clients report connection resets, your 5xx rate ticks up, and server.hot_restart_epoch is climbing on your dashboard. The new process was supposed to inherit listen sockets gracefully while the old one drained.

Hot restart is Envoy’s mechanism for zero-downtime binary upgrades and certain reloads. A new process launches, coordinates with the old one over a Unix domain socket, takes over the listen sockets, and the old process enters a drain sequence. Both processes run simultaneously during the handoff. That coexistence is where the races live.

The symptoms are specific: dropped connections during rollover, server.hot_restart_epoch incrementing in rapid succession (a crash loop that looks like a hot restart), or anomalous stats values after a version upgrade. The root causes are mechanical: file descriptor exhaustion from the dual-process window, the previous epoch dying before the handoff completes, a concurrency decrease that drops accept-queue connections, or stats schema changes that corrupt shared state.

What this means

Hot restart is a two-process coordination protocol. The old process (epoch N) is serving traffic. A new process (epoch N+1) starts and connects to the old process over a Unix domain socket. The new process sends a restart RPC. The old process hands off its listen sockets, indexed by worker, and transitions to server.state = 1 (DRAINING). The new process binds the inherited sockets, loads its initial xDS configuration, warms its clusters and listeners, and transitions to server.state = 0 (LIVE).

sequenceDiagram
    participant Old as Old (epoch N)
    participant UDS as UDS
    participant New as New (epoch N+1)

    Old->>UDS: listening for restart RPC
    New->>UDS: connect, send restart RPC
    UDS->>Old: forward RPC
    Old->>Old: state = DRAINING
    Old->>New: hand off listen sockets
    Note over Old,New: Both running: FDs double, race window
    New->>New: bind sockets, load xDS
    New->>New: state = LIVE
    Old->>New: transfer stats
    Old->>Old: drain connections
    Old->>Old: parent_connections to 0
    Old->>Old: shutdown

The dangerous window is between “old process starts draining” and “new process is fully LIVE.” During this period:

• Both processes hold open file descriptors. Total FD usage briefly doubles. If baseline FD usage exceeds 50% of the limit, the dual-process window can push past the FD ceiling and cause silent connection refusal.
• The old process is draining: it stops accepting new connections on some listeners and sends Connection: close on HTTP/1.1 connections and GOAWAY on HTTP/2 streams. It still holds in-flight connections open until they complete or the drain time expires.
• The new process may still be initializing (waiting for xDS, warming clusters). If it has not reached LIVE, it is not accepting new connections on all listeners either.

If the old process finishes draining before the new process reaches LIVE, or if the old process crashes mid-drain, there is a gap where neither process serves traffic. Connections in the kernel’s accept queue or in-flight on the old process can be dropped.

On Linux, Envoy defaults to SO_REUSEPORT sockets, which lets both processes bind the same port simultaneously. During hot restart, sockets are passed by worker index so the kernel distributes new connections correctly. But if --concurrency decreases between epochs, some workers in the old process have no counterpart in the new process, and connections queued on those orphaned workers can be dropped.

Hot restart is enabled by default and is not supported on Windows. In Istio sidecar mode, the pilot-agent starts Envoy with --disable-hot-restart, so this mechanism does not apply unless you have overridden that flag.

Common causes

Quick checks

These commands are read-only and safe to run during an incident. Adjust the admin port for your deployment (default 9901, or 15000 in Istio sidecar mode).

# Check process state, epoch, and parent connections
curl -s http://localhost:9901/stats | grep -E 'server\.(state|hot_restart_epoch|live|parent_connections)'

# Check readiness: 200 means LIVE, 503 means draining or initializing
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:9901/ready

# Get server info including epoch and uptime
curl -s http://localhost:9901/server_info | python3 -m json.tool

# Check FD usage on the Envoy process
ENVOY_PID=$(pgrep -x envoy | head -1)
ls /proc/$ENVOY_PID/fd | wc -l
grep 'Max open files' /proc/$ENVOY_PID/limits

# Check total connections across both processes
curl -s http://localhost:9901/stats | grep 'server.total_connections'

# Check hot restart version compatibility
curl -s http://localhost:9901/hot_restart_version

# Count Envoy processes (2 during hot restart, 1 otherwise)
pgrep -x envoy | wc -l

How to diagnose it

1. Confirm a hot restart is in progress or recently completed. Check server.hot_restart_epoch. If it has incremented in the last few minutes, a hot restart occurred. Check pgrep -x envoy. If two processes are running, the handoff is still in progress.

2. Determine whether the old process is still draining. Check server.parent_connections. A non-zero value means the old process still holds connections and has not finished draining. If server.parent_connections stays non-zero longer than --drain-time-s (default 600 seconds), the drain is stuck.

3. Check for a crash loop. If server.hot_restart_epoch increments more than once per minute, the new process is crashing and the supervisor is restarting it. Look at Envoy stderr logs for the crash reason. Common causes: bad xDS config that fails validation, incompatible binary, or an assert failure in the hot restart RPC path.

4. Check FD pressure. During the dual-process window, FD usage doubles. Run ls /proc/<pid>/fd | wc -l against both PIDs and compare against the limit from /proc/<pid>/limits. If usage is above 80% of the limit, FD exhaustion is the likely cause of connection drops.

5. Verify the new process reached LIVE. Check server.state on the new process (should be 0). If it is stuck at 2 or 3 (PRE_INITIALIZING or INITIALIZING), the new process is waiting for xDS configuration. The old process eventually shuts down after --parent-shutdown-time-s (default 900 seconds). If the new process has not reached LIVE by then, you get a gap with no serving process.

6. Check for stats anomalies after upgrades. If the hot restart crossed a version boundary, compare stats output before and after. Look for missing counters, impossible gauge values (negative connection counts, gauges that should be monotonic going backwards), or counters that reset unexpectedly. The stats transfer between old and new processes can produce corrupt values when the stat schema differs between versions.

7. Check restart trigger frequency. If you use SIGHUP or a restart script, look at how frequently it fires. The hot restart protocol has a known race at very short restart intervals (Envoy issue #550). Restarts triggered within roughly 250ms of each other can kill both parent and child simultaneously.

Metrics and signals to monitor

Fixes

Crash loop via hot restart

If the new process crashes immediately after starting, the hot restart protocol cycles. Each failed epoch leaves behind state and compounds the problem.

Check Envoy stderr logs for the crash reason. If the crash is from bad xDS config, fix the config or roll back the control plane change. If the crash is from a binary incompatibility, verify version compatibility using --hot-restart-version on the new binary and compare against GET /hot_restart_version on the running process. If the versions are incompatible, perform a cold restart (stop the old process, start the new one) instead of a hot restart.

For environments where the parent process may die before the child initializes, consider --skip-hot-restart-on-no-parent. Without this flag, the child terminates if the parent is gone, which can create a loop if the parent is unstable.

FD exhaustion during dual-process window

The simplest fix is to raise the FD limit (ulimit -n or the container security context). For production proxies, an FD limit of at least 65536 is reasonable. The deeper fix is to reduce baseline FD usage so the dual-process window does not push past the limit: shorten idle timeouts to close stale connections faster, verify connection pooling is enabled, and check for FD leaks from access log files or excessive health check connections.

Keep baseline FD usage below 50% of the limit when hot restart is in use. The restart temporarily doubles it.

Previous epoch dies before handoff completes

If the old process crashes or is killed (OOM, signal, orchestrator eviction) before the new process has sent its restart RPC and received the listen sockets, the hot restart fails. The new process either asserts or falls back depending on flags.

Check the old process exit code and logs. If it was OOM-killed, address memory pressure. If it was killed by the orchestrator because the termination grace period was too short, increase the grace period to exceed --drain-time-s plus a buffer.

--skip-hot-restart-on-no-parent allows the new process to fall back to a normal startup if the parent is gone, instead of terminating. This trades the zero-downtime property for resilience.

Concurrency decrease drops connections

If --concurrency decreases between epochs, the old process has more workers than the new one. Connections queued on workers that have no counterpart in the new process are dropped because no worker inherits that socket.

This is a known limitation of the socket-handoff mechanism. To avoid it, do not decrease concurrency during a hot restart. If you must reduce worker count, do it as a separate cold restart after the hot restart completes, or accept the connection drops during the transition.

Stats corruption after version upgrade

When the old and new processes run different Envoy versions, the stats schema may differ. The stats transfer between processes can produce corrupted or missing values. Watch for anomalous stats after upgrades, because the stat schema can change across versions.

Envoy transfers stats between hot-restart processes as protobuf messages over the Unix domain socket rather than via a fixed-size shared memory region (change merged in PR #5910, shipped in Envoy 1.11.0). This eliminated the shared-memory size mismatch crash that occurred when upgrading across major versions. However, schema differences between versions can still produce anomalous values during the transfer.

The safest approach for major version upgrades is to skip hot restart entirely and perform a cold restart. For minor version upgrades within the same hot restart compatibility version (check --hot-restart-version), hot restart should be safe. Monitor stats output for anomalies after any upgrade that crosses a version boundary.

--skip-hot-restart-parent-stats disables stats import from the parent process entirely. This prevents corruption but means counters reset to zero on each restart, losing continuity.

Note: a cold restart (SIGTERM or /quitquitquit) does not perform graceful draining the way hot restart does. TCP resets occur on shutdown rather than the orderly Connection: close and GOAWAY sequence. Plan for client-visible disruption if you choose this path.

Prevention

• Rate-limit restart triggers. The hot restart protocol races at very short intervals. Ensure your restart mechanism (SIGHUP handler, deployment script, supervisor) does not trigger more than one restart per second, and ideally no more than one per several seconds.
• Size FD limits for the dual-process window. Calculate your limit as roughly baseline_peak_FD * 2.5 to account for the doubling during restart plus headroom.
• Verify hot restart compatibility before upgrading. Compare --hot-restart-version of the new binary against the running process via the admin endpoint. If they differ, plan a cold restart.
• Keep --parent-shutdown-time-s larger than --drain-time-s. The parent must survive long enough for the new process to reach LIVE. Default values (600s drain, 900s parent shutdown) provide a 300-second buffer. If your xDS initialization is slow, increase the parent shutdown time.
• Use --skip-hot-restart-on-no-parent in unstable environments. If the parent process is prone to being killed (short Kubernetes termination grace periods, aggressive OOM killer), this flag prevents the child from terminating when the parent disappears.
• Monitor stats after version upgrades. Compare key gauges and counters before and after any binary upgrade. If values look wrong, restart cold to reset the stats subsystem.
• In Istio sidecar mode, do not send SIGHUP to Envoy. Istio starts Envoy with --disable-hot-restart. SIGHUP is not a supported restart mechanism in this mode. Use the pod lifecycle instead.

How Netdata helps

• Per-second server.hot_restart_epoch collection detects crash loops within seconds of the first failed epoch, before the loop compounds.
• Correlating server.state transitions with connection count changes across the restart window identifies whether drops are from the drain sequence, FD exhaustion, or a gap between processes.
• server.parent_connections tracking distinguishes a stuck drain from a failed handoff by showing whether the old process is still draining or has exited prematurely.
• FD utilization monitoring against the configured limit catches the dual-process doubling before it causes silent connection refusal.
• Anomaly detection on stats values after a version upgrade surfaces corrupted or impossible gauge values that would otherwise go unnoticed until they trigger a false alert elsewhere.
• Combining server.uptime resets with epoch increments in a single timeline distinguishes a crash-loop-via-hot-restart from a planned rolling deployment.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy clusters stuck warming: warming_clusters non-zero and routes returning 503
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy control_plane.connected_state = 0: running on stale xDS config
• Envoy downstream 4xx spike: 401s, 403s, and 404s from the client side
• Envoy downstream connection flood: slowloris, the cx-to-rq ratio, and oversized requests
• Envoy downstream_cx_active growing: connection leaks and idle-timeout gaps
• Envoy downstream_cx_overflow and overload_reject: connections turned away at the door
• Envoy downstream_rq_time high: client-observed latency and proxy overhead