Envoy rate limiting: over_limit, 429s, and fail-open vs fail-closed

Envoy has two HTTP filters for rate limiting with very different failure characteristics. The global rate limit filter delegates every applicable request to an external rate limit service (RLS) over gRPC. The local rate limit filter applies an in-process token bucket with no external dependency. Both can produce 429 responses, but the signals that tell you what happened live in different stat namespaces and mean different things.

The harder operational question is not “are we rate limiting” but “what happens when the rate limit service itself is down.” That answer is controlled by one setting: failure_mode_deny. Its default is false (fail-open), meaning a broken RLS silently disables rate limiting. Fail-closed (failure_mode_deny: true) instead rejects every request through the filter. A misread here is a common source of incidents.

What it is and why it matters

Two filters, similar names, different mechanics:

• Global rate limit filter (envoy.filters.http.ratelimit): on every request that matches a rate limit action, Envoy makes a synchronous gRPC call to an external RLS. The RLS evaluates descriptors derived from the request and returns OK or OVER_LIMIT for each. This filter produces the ok, over_limit, error, and failure_mode_allowed counters. It is the only one with fail-open vs fail-closed behavior, because it is the only one that depends on an external service.
• Local rate limit filter (envoy.filters.http.local_ratelimit): applies a token bucket inside the Envoy process. No external service, no network call, no fail-open decision. Stats live under http_local_rate_limit.*.

The failure modes are completely different. The local filter either has tokens or it does not. The global filter can fail in three ways: the RLS says OK, the RLS says OVER_LIMIT, or the RLS is unreachable. The third case is where operators get burned, because the outcome depends on failure_mode_deny, a single field that is easy to overlook.

How it works

The global rate limit filter

When a request matches a route or virtual host with a rate limit action configured, the global filter builds descriptors from request attributes (source IP, path, headers, and so on) and sends them to the RLS. The RLS responds per descriptor. If any descriptor returns OVER_LIMIT, the request is rejected. The response drives which counter increments and what Envoy returns to the client.

The four counters that matter:

The stat prefix depends on filter configuration. Counters appear under http.<stat_prefix>.ratelimit.* where stat_prefix is set on the filter. Grep for ratelimit on the admin endpoint to find them:

curl -s http://localhost:9901/stats | grep ratelimit

When the RLS returns OVER_LIMIT, Envoy sends a 429 by default. The response code is configurable via rate_limited_status; values below 400 are clamped to 429. Envoy sets the x-envoy-ratelimited header on the 429 response unless disable_x_envoy_ratelimited_header is set. For gRPC traffic, the default maps rate-limited calls to the UNAVAILABLE status code. Setting rate_limited_as_resource_exhausted: true switches this to RESOURCE_EXHAUSTED, which is what most gRPC clients expect for rate limiting.

flowchart TD
    A[Request hits global rate limit filter] --> B[Call external RLS]
    B --> C{RLS response}
    C -->|OK| D[Forward to upstream]
    C -->|OVER_LIMIT| E["429, over_limit++"]
    C -->|error or timeout| F{failure_mode_deny}
    F -->|"false: fail-open"| G["Allow, failure_mode_allowed++"]
    F -->|"true: fail-closed"| H["Reject, status_on_error 500"]

The local rate limit filter

The local filter applies a token bucket inside the Envoy process. No external service, no network call. The bucket has a max token count, a tokens-per-fill value, and a fill interval. The token bucket’s fill_interval must be at least 50ms to avoid overly aggressive refills; smaller values are rejected.

Stats live under http.<stat_prefix>.http_local_rate_limit.*: enabled (requests the limiter consulted), ok, rate_limited, and enforced. The distinction between rate_limited and enforced matters. rate_limited counts requests the token bucket rejected. enforced reflects the filter_enforced runtime fraction. If enforcement is set below 100%, some requests that would be rejected are still forwarded, and the two counters diverge.

The critical property of the local filter is that the token bucket is per-worker. Envoy’s worker threads share nothing in the hot path. A limit of 100 rq/s configured on an Envoy with 8 workers allows 800 rq/s total, not 100. If you need a hard global cap, use the global filter with an external RLS, or account for worker count when sizing local limits.

Fail-open vs fail-closed: the failure_mode_deny decision

The global filter’s behavior when the RLS is unreachable is controlled by failure_mode_deny on the filter configuration. The default is false (fail-open).

Fail-open (failure_mode_deny: false, the default): when the RLS call fails, Envoy allows the request through and increments failure_mode_allowed. Rate limiting is effectively disabled for the duration of the outage. This protects availability: a broken RLS does not take down your service. The cost is that any protection the rate limiter was providing (abuse mitigation, tenant isolation, upstream protection) is gone for as long as the RLS is down.

Fail-closed (failure_mode_deny: true): when the RLS call fails, Envoy rejects the request. The response code is controlled by status_on_error, which defaults to 500, not 429. This protects the upstream: if you cannot confirm a request is within limits, you do not let it through. The cost is that an RLS outage becomes a full outage for every route that goes through the filter.

Both are legitimate choices. The mistake is not knowing which one you have. Check the running config to confirm:

curl -s http://localhost:9901/config_dump | grep -A5 -B5 failure_mode_deny

The error and failure_mode_allowed counters tell you which mode is active in real time. During an incident, the difference between “rate limiting is silently off” and “all traffic is rejected with 500s” is one configuration field.

The failure_mode_deny_percent field (added in 1.35.0) allows a runtime-fraction-based mix. With failure_mode_deny: true and failure_mode_deny_percent at 50%, roughly half of requests are denied during an RLS outage and the rest pass through — a load-shedding middle ground for deployments that want partial protection without a hard cliff. If the runtime key is not set, the field’s default value applies; if the field is unset entirely, plain failure_mode_deny behavior wins.

Where it shows up in production

429 spikes that are working as designed

A rising over_limit rate is not necessarily a problem. It means the rate limiter is doing its job. A product launch, a misconfigured client retrying in a tight loop, or a downstream service suddenly doubling its call rate can all produce legitimate over_limit spikes. over_limit needs business context to interpret.

What is worth alerting on is a sudden change in the over_limit / (ok + over_limit) ratio, especially without a corresponding traffic increase. That suggests either a configuration change tightened limits or a client changed behavior.

Error spikes: the RLS is unhealthy

A non-zero error rate means the RLS call is failing. This is the signal that triggers the fail-open vs fail-closed decision. Compute error / (ok + error + over_limit): sustained values above 1% indicate the RLS is unreliable.

Correlate error with failure_mode_allowed. If both are climbing, you are in fail-open mode and rate limiting is disabled. If error is climbing and failure_mode_allowed stays at zero while 500s rise, you are in fail-closed mode and traffic is being rejected.

The per-worker token bucket surprise

The most common local rate limit incident is not a failure but a misreading of capacity. An operator configures a local rate limit of 500 rq/s, deploys to an Envoy running 16 workers, and discovers the actual limit is 8000 rq/s. The upstream they were protecting gets many times the intended load. Always multiply the configured local rate by the worker count (--concurrency flag; the server.concurrency gauge on the admin endpoint, also visible in /server_info command-line options) when reasoning about effective throughput.

gRPC clients and the status code

For gRPC workloads, the status code Envoy returns on OVER_LIMIT determines whether clients back off correctly. UNAVAILABLE, the default, is often treated as a transient error and retried aggressively, which is the opposite of what rate limiting intends. RESOURCE_EXHAUSTED signals to the client that it should slow down. If your gRPC clients honor RESOURCE_EXHAUSTED but not UNAVAILABLE, set rate_limited_as_resource_exhausted: true.

Signals to watch in production

Common misuses

• Not knowing your failure_mode_deny setting. During an RLS outage, this single field is the difference between “rate limiting off” and “all traffic rejected.” The default is false (fail-open). Verify it in config and monitor failure_mode_allowed.
• Alerting on raw over_limit counts. over_limit is often working as designed. Alert on ratio changes, not absolute counts.
• Treating local rate limits as global caps. The token bucket is per-worker. A 100 rq/s limit on an 8-worker Envoy is 800 rq/s.
• Forgetting the gRPC status code. Default UNAVAILABLE may cause client retry storms. Consider rate_limited_as_resource_exhausted for gRPC traffic.
• Running the RLS as a single point of failure. If the RLS is down, every route through the global filter is affected. The RLS needs its own redundancy and health monitoring.

How Netdata helps

• Correlate ratelimit.error and ratelimit.failure_mode_allowed against the RLS cluster’s own health (membership_healthy, upstream_rq_5xx) to confirm whether an error spike is the RLS failing or Envoy losing connectivity to it.
• Track over_limit, ok, error, and failure_mode_allowed as per-second rates with anomaly detection so a shift in the ratio surfaces even when absolute counts look normal.
• For local rate limiting, chart http_local_rate_limit.rate_limited alongside Envoy worker count to make the per-worker effective limit visible at a glance.
• Pair 429 response rates with the rate limit counters to distinguish rate-limiter rejections from auth-driven or upstream-driven 4xx.
• Use the RLS cluster’s own latency histograms (upstream_rq_time) to catch a slow RLS before it trips timeouts and floods the error counter.

Related guides

• Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures
• Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend
• Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag
• Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests
• Envoy clusters stuck warming: warming_clusters non-zero and routes returning 503
• Envoy connection pool exhaustion: a slow upstream that fills the pool
• Envoy control_plane.connected_state = 0: running on stale xDS config
• Envoy downstream 4xx spike: 401s, 403s, and 404s from the client side
• Envoy downstream connection flood: slowloris, the cx-to-rq ratio, and oversized requests
• Envoy downstream_cx_active growing: connection leaks and idle-timeout gaps
• Envoy downstream_cx_overflow and overload_reject: connections turned away at the door
• Envoy downstream_rq_time high: client-observed latency and proxy overhead