$ guides / envoy / envoy-listener-create-failure ▌

Operations Guides

Envoy listener_create_failure: a listener config Envoy could not apply

The listener_manager.listener_create_failure counter increments when Envoy cannot add a listener object to its workers. In a healthy deployment this counter is zero. Any non-zero value means Envoy received a listener configuration it could not apply: a failed bind (port conflict, permission denied) or invalid configuration.

The signal is easy to miss because Envoy does not crash. It keeps serving the previous listener configuration, traffic on existing routes flows normally, and the listener the operator intended to add or modify never becomes active. Deployment pipelines report success. The control plane believes the config was accepted. Only the process log and this counter reveal the failure.

Treat this as the “connected but rejecting” variant of a silent NACK. control_plane.connected_state is 1. The xDS stream is up. But listener_create_failure climbing alongside update_rejected means Envoy is silently discarding new configuration. Teams that miss this counter discover the problem only when a new route or a certificate rotation silently fails to take effect.

What this means

A listener in Envoy is the entry point for downstream traffic: a bound socket with an address, port, and filter chain. When the listener manager receives a new or updated listener configuration (via static config, LDS, or delta-xDS), it attempts to create listener objects on each worker thread. If any step fails, listener_create_failure increments.

The two broad failure classes are:

Bind failures: Envoy cannot bind the configured address. Causes include a port already in use by another process, permission denied on privileged ports (below 1024) when running as a non-root user, or unsupported socket options on the platform.
Configuration conflicts: The listener conflicts with an existing one. Listeners are identified by name for LDS update semantics, but two listeners cannot bind the same address:port. A static listener and an LDS-provided listener sharing the same address will conflict at bind time even if their names differ.

Envoy does not exit on bind failure. It continues running with the failed listener in a non-serving state. The listener may still appear in the /listeners admin endpoint, which can mislead operators into thinking it is active. Traffic intended for the failed listener may silently match a different listener, such as a wildcard listener on the same port.

Common causes

Cause	What it looks like	First thing to check
Privileged port bind as non-root	`listener_create_failure` increments immediately after pod startup; listener on port below 1024 never accepts traffic	Envoy container user and the listener port
Port already in use	Failure during a config update, not at startup; another process holds the port	`ss -ltnp` on the host or inside the pod network namespace
Static vs LDS address collision	Failure correlates with an LDS push; static listener works, LDS listener rejected	Compare addresses and names in static config vs the LDS response
Invalid socket option	Failure on platforms that do not support the requested option (for example, `freebind` on macOS)	Process log for the specific socket option error
Invalid filter chain config	Failure after an xDS push; rejection logged with a config validation error	Envoy stderr for the validation message

Quick checks

These assume the Envoy admin endpoint is reachable from where you are running them. Do not expose the admin port externally; reach it via kubectl exec, nsenter, or a local port-forward.

# Check the counter value and rate
curl -s http://localhost:9901/stats | grep listener_create_failure

# Confirm control plane is connected (the "connected but rejecting" pattern)
curl -s http://localhost:9901/stats | grep -E 'connected_state|update_rejected'

# List active listeners - a failed listener may still appear here
curl -s http://localhost:9901/listeners

# Check warming listeners - a stuck listener may sit in warming
curl -s http://localhost:9901/stats | grep -E 'total_listeners_warming|total_listeners_active'

# Dump the current listener config to verify what Envoy actually applied
curl -s http://localhost:9901/config_dump | jq '.configs[] | select(.type_url | contains("listener"))'

# Check what process owns the conflicting port
ss -ltnp | grep ':<port>'

# Check the Envoy process user (relevant for privileged port binds)
id

In sidecar deployments (Istio, Consul Connect), the admin port is typically 15000 rather than 9901. Adjust accordingly.

How to diagnose it

flowchart TD
    A[listener_create_failure climbing] --> B{connected_state?}
    B -->|0| C[Fix xDS connection first]
    B -->|1| D{update_rejected climbing?}
    D -->|Yes| E[Broader config rejection]
    D -->|No| F[Listener-specific failure]
    F --> G[Read process log]
    G --> H{Failure class?}
    H -->|Permission denied| I[Privileged port as non-root]
    H -->|Address in use| J[Port conflict]
    H -->|Validation error| K[Invalid config]
    H -->|Socket option| L[Unsupported platform feature]

Confirm the counter is actually incrementing. Take two samples a few seconds apart. A static non-zero value left over from a past incident is less urgent than an actively climbing counter.
Check control_plane.connected_state. If it is 0, the issue is upstream of the listener manager. Fix the xDS connection first. If it is 1 and update_rejected is also climbing, you have the connected-but-rejecting pattern.
Read the Envoy process log. The stats endpoint tells you THAT a listener failed, not WHY. The rejection reason is logged to stderr but not exposed via stats. Look for lines mentioning the listener name and address.
Identify the specific listener. Cross-reference the failing listener name from the log with the LDS or static config. The failure is per-listener, not global.
Determine the failure class from the log message:
- “permission denied” or “operation not permitted”: privileged port issue.
- “address already in use”: port conflict.
- A config validation error: invalid configuration.
- A socket option name: unsupported platform feature.
Verify the fix took effect. After correcting the issue, watch listener_create_failure stop increasing and listener_manager.total_listeners_active increase. The listener transitions out of warming.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`listener_manager.listener_create_failure`	The primary signal; increments on every rejected listener add	Any non-zero value, especially when climbing
`listener_manager.total_listeners_warming`	Listeners received but not yet active; a stuck listener sits here	Non-zero sustained outside of a config push window
`listener_manager.total_listeners_active`	Count of fully active listeners; should match expected count	Drops, or fails to increase after a deployment
`control_plane.connected_state`	Distinguishes “disconnected” from “connected but rejecting”	0 means xDS is down; 1 with rejection climbing means silent NACK
`cluster.<name>.update_rejected`	Parallel signal for cluster config rejections	Non-zero indicates broader config rejection, not just listeners
`server.state`	Process lifecycle state; confirms Envoy is LIVE	Non-zero (DRAINING or INITIALIZING) during normal operation

Fixes

Privileged port bind as non-root

The official Envoy Docker images run as non-root by default. Binding ports below 1024 requires elevated privileges. Three options:

Set ENVOY_UID=0 in the container environment to run Envoy as root. This restores older behavior but broadens the blast radius of any compromise.
Grant CAP_NET_BIND_SERVICE to the container. This is the narrower fix. Envoy can bind privileged ports without running as root.
Use a port above 1024. Often the cleanest fix if the upstream infrastructure (load balancer, Kubernetes Service) can be adjusted.

Pick one and verify with a restart. Do not combine these blindly.

Port already in use

Identify the conflicting process with ss -ltnp or lsof -i :<port>. Common causes in production:

A previous Envoy process did not fully drain during hot restart and still holds the socket.
A sidecar injector or init container bound the same port.
Consul or Nomad static port allocations collide with dynamically allocated ports.

The fix is to either stop the conflicting process or change the listener port. If the conflict is from a zombie Envoy process during hot restart, verify server.parent_connections drains to zero.

Static vs LDS address collision

Two listeners cannot bind the same address:port. If a static listener and an LDS-provided listener target the same address, the second one is rejected regardless of name. The fix is one of:

Remove the static listener and serve it entirely via LDS.
Use a different address or port for the LDS listener.

Check the LDS response and the static config side by side to confirm the collision.

Invalid socket options

The freebind listener option (freebind: true) is not supported on all platforms. On macOS in particular, setting freebind causes listener creation to fail. On Linux it is supported. If the log indicates a socket option failure, either remove the option or restrict its use to platforms that support it.

Invalid filter chain configuration

If the rejection is due to invalid configuration (a bad filter, an unsupported field, a schema validation error), the fix is in the config source: the control plane or the static config. The Envoy log will contain the specific validation error. Common causes include:

Referencing a cluster that does not exist.
Using a filter configuration unsupported by the installed Envoy version.
Schema validation failures from version skew between the control plane and Envoy.

Prevention

Alert on listener_create_failure being non-zero. This counter should always be zero. Treat any non-zero value as a config or control-plane bug.
Alert on the connected-but-rejecting combination. Page or ticket when control_plane.connected_state = 1 AND (update_rejected OR listener_create_failure) is climbing. This catches silent NACKs that connected_state alone misses.
Pin the Envoy container user explicitly. If you depend on privileged port binds, set ENVOY_UID or grant CAP_NET_BIND_SERVICE in your deployment manifest, not at debug time.
Validate listener names and addresses across static and dynamic config. Before adding an LDS listener, confirm its address does not collide with a static listener.
Run config validation in CI. Run envoy --mode validate against config before pushing to production. This catches schema errors before they reach a live Envoy.
Monitor total_listeners_active against expected count. If you expect N listeners and have fewer, something failed to apply.

How Netdata helps

Netdata collects the listener manager and control plane counters at per-second resolution, which lets you correlate three signals in one view during a config push:

listener_manager.listener_create_failure as a per-second rate makes a climbing rejection immediately visible rather than buried in a longer scrape interval.
control_plane.connected_state next to the rejection counters distinguishes “disconnected” from “connected but rejecting.”
listener_manager.total_listeners_warming and total_listeners_active show whether a failed listener is stuck warming or never registered.
update_rejected on clusters alongside the listener counters reveals whether the rejection is listener-specific or part of a broader config-push problem.