$ guides / uwsgi / uwsgi-connection-pool-cascade ▌

Operations Guides

uWSGI connection pool cascade: downstream latency that stalls every worker

Workers climbing toward all-busy. Response times creeping up. Throughput falling. No traffic spike, no deploy, no code change. The pattern built over minutes, not seconds.

The mechanism is a reinforcing feedback loop. A downstream dependency (database, cache, external API) gets slower. Not dead, just slower. Each request now holds its downstream connection longer. The pool fills. The next request that needs a connection blocks waiting for one to be returned. That wait adds to the request’s wall-clock duration, which keeps the worker busy longer, which keeps the connection occupied longer. The loop tightens until every worker is blocked on pool checkout and throughput collapses.

uWSGI metrics show the symptoms (busy ratio, response time, exceptions) but not the root cause. uWSGI does not expose downstream connection pool depth. The pool fill is invisible unless you instrument the application’s database client or query the downstream system directly.

Anatomy of the cascade

The connection pool cascade is distinct from other uWSGI saturation patterns. In a traffic spike, busy ratio jumps suddenly and throughput stays high. In a harakiri death spiral, the downstream is completely dead and every request hits the timeout ceiling. In a pool cascade, the downstream is merely slow, workers are merely waiting, and the system grinds to a halt through accumulated holding time.

Each uWSGI worker in pre-fork mode runs a full copy of the application. If the application uses a per-process connection pool (common with SQLAlchemy, psycopg2, or similar database clients), total downstream connection demand is num_workers x pool_size_per_worker. When the downstream gets slower, each connection is held longer per request. The pool has a fixed number of slots. Once all are checked out, new requests block.

flowchart TD
    A["Downstream latency rises"] --> B["Workers hold connections longer"]
    B --> C["Connection pool fills"]
    C --> D["New requests block on pool checkout"]
    D --> E["Request wall time increases"]
    E --> F["Workers stay busy longer"]
    F -->|"reinforcing loop"| B
    D --> G["Busy ratio climbs gradually"]
    G --> H["All workers blocked"]
    H --> I["Throughput stalls"]

The critical relationship is between pool size, connection holding time, and request rate. If your pool has 5 connections per worker, downstream queries average 50ms, and each worker handles 100 req/s, each connection serves about 20 req/s. At 50ms per query, 5 connections sustain 100 req/s. But if downstream latency rises to 200ms, those same 5 connections sustain only 25 req/s per worker. The pool fills, and requests start blocking on checkout.

Distinguishing marks:

Busy ratio climbs gradually over minutes, not seconds. This is the key tell. A traffic spike or a harakiri storm produces a sharp step. The cascade produces a steady ramp.
Exception messages mention pool timeouts or “too many connections.” The application is failing to acquire a downstream connection, not failing to process the request itself.
The downstream system reports its connection count at or near maximum. Not refusing connections yet, but no room for more.
Harakiri may eventually trigger, but the cascade starts before any timeout fires. Workers are blocked on pool checkout, not on the downstream call itself.

Common causes

Cause	What it looks like	First thing to check
Downstream latency increase	avg_rt rises before busy ratio climbs; downstream query times elevated	Downstream service latency directly (slow query log, cache hit ratio, API response time metrics)
Connection pool too small	Pool always near full even at normal downstream latency; workers block on checkout during traffic bursts	Application-side pool utilization (checked-out vs. total connections)
Connection leak	Pool fills monotonically over time; recycling workers temporarily clears it, then it refills	Compare checkout count against checkin count; inspect exception paths that skip connection release
Aggregate demand exceeds downstream limit	Multiple app servers each with their own pool; downstream reports max connections reached	Sum `num_workers x pool_size` across all app servers; compare against downstream max_connections

Quick checks

All commands are read-only and safe during an incident. Replace 127.0.0.1:9191 with your stats server address and PORT with your listening port.

# Worker busy ratio - compare against the last 10-30 minutes, not a single point
uwsgi --connect-and-read 127.0.0.1:9191 | jq '([.workers[] | select(.status == "busy")] | length) as $busy | ([.workers[] | select(.pid > 0 and .status != "cheap")] | length) as $alive | if $alive > 0 then ($busy / $alive * 100) else 0 end'

# Average response time (microseconds) - EMA, not a cumulative average
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.pid > 0 and .status != "cheap")] | if length > 0 then (map(.avg_rt) | add / length) else 0 end'

# Exception count (monotonic) - look for pool or connection errors in app logs
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].exceptions] | add'

# Stuck request ages - shows how long each in-flight request has been running
uwsgi --connect-and-read 127.0.0.1:9191 | jq --argjson now "$(date +%s)" '[.workers[] | select(.pid > 0) | .id as $wid | .cores[] | select(.in_request == 1) | {worker: $wid, core: .id, age_seconds: ($now - .req_info.request_start)}]'

# Socket queue depth (external) - uWSGI's listen_queue field is unreliable on standard Linux
ss -ltn 'sport = :PORT'

# Harakiri count (per-worker, monotonic)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].harakiri_count] | add'

On socket queue monitoring: the listen_queue and load fields in uWSGI stats output are unreliable on standard Linux. The TCP measurement via TCP_INFO varies across kernel versions, and the load field is identical to listen_queue (not average latency, despite its name). The listen_queue_errors field exists in the JSON but is not incremented by the source code. Use ss -ltn for TCP or ss -lxn for UNIX sockets, and check the Recv-Q column for actual queue depth.

How to diagnose

Confirm the gradual climb. Compare current busy ratio against the trend over the last 10 to 30 minutes. A cascade shows a steady upward ramp. A traffic spike produces a step function. This single distinction rules out most other causes of all workers busy.
Check downstream connection utilization. This is the signal uWSGI cannot give you. Query the downstream directly. For PostgreSQL, check pg_stat_activity. For MySQL, SHOW STATUS LIKE 'Threads_connected'. For Redis, INFO clients. Compare against the downstream’s configured max connection limit.
Inspect exception messages. Look for pool checkout timeouts, “too many connections,” or connection wait exceeded errors. These confirm that workers are failing to acquire connections, not failing to process requests. Note that uWSGI’s exception counter (workers[].exceptions) captures only exceptions that bubble up to the WSGI layer. Application-level try/except blocks that handle errors internally do not increment this counter.
Check stuck request ages. If workers are blocked on pool checkout, the per-core in_request timestamp shows requests running far longer than the downstream query itself takes. The gap between request age and actual downstream latency is the pool wait time. If --stats-no-cores is enabled, this signal is unavailable.
Determine leak vs. demand. If pool utilization is high even at low traffic with normal downstream latency, suspect a connection leak (connections borrowed but never returned). If it tracks with traffic volume and downstream latency, it is genuine demand exceeding pool capacity.
Check for retry amplification. When a client times out waiting for a response, it may retry. The retry demands another connection from the already-exhausted pool. Check client-side retry rates and timeout settings.
Check avg_rt trend against harakiri. avg_rt is an EMA computed as (old_avg_rt + current_request_time) / 2, so each new request contributes 50% of the value. After roughly 7 requests, older contributions are negligible. If avg_rt is climbing toward your configured harakiri timeout, workers are about to start dying.

Metrics to monitor

Signal	Why it matters	Warning sign
Worker busy ratio	Primary capacity utilization metric	Gradual climb above 80% sustained, approaching 100%
avg_rt (microseconds)	EMA-weighted response time trend	Sustained increase above 2x recent baseline
Exception rate	Pool exhaustion surfaces as WSGI-layer exceptions	Rising rate with pool or connection error messages in logs
Stuck request age	Shows how long workers have been blocked	Request ages far exceeding normal downstream query latency
Downstream connection count	Root cause indicator	At or near the downstream system’s max_connections limit
Harakiri count	Late-stage signal	Any sustained non-zero rate means requests are timing out
Socket Recv-Q (via ss)	Kernel-level saturation indicator	Sustained non-zero value means connections backing up

Fixes

Emergency: reduce worker count

Warning: this reduces throughput immediately. Use only to stabilize while investigating root cause.

Total connection demand equals num_workers x pool_size_per_worker. Reducing workers cuts total demand immediately, giving the pool room to drain. If you are running the cheaper subsystem, lower the processes maximum to cap total connection demand.

This is a stopgap, not a fix.

Downstream latency is the cause

Investigate the downstream service directly. Common culprits: slow queries (check slow query logs), missing indexes, lock contention, cache miss storms, or the downstream service itself being overloaded.

Set per-query or per-call timeouts shorter than your harakiri timeout. If a query takes 5 seconds and your harakiri is 30 seconds, the worker holds the connection for 5 seconds. Cap the query at 2 seconds and fail fast, and the connection returns to the pool sooner, increasing effective pool throughput.

Pool is too small

Increase the per-worker pool size, or reduce the worker count so that num_workers x pool_size fits within the downstream’s connection limit with headroom.

The constraint is bidirectional. Each worker needs enough connections to handle its concurrent request load without blocking. But the aggregate across all workers must not exceed what the downstream can serve. If you have 8 app servers with 16 workers each and a 5-connection pool, that is 640 connections. If the database max_connections is 200, you are already oversubscribed.

Connection leak

Connections are borrowed from the pool but never returned, typically due to exception paths that skip the release call. Symptoms: pool fills monotonically over time, recycling workers temporarily clears it (because the pool is reinitialized on fork), then it refills.

Fix the code path. Use context managers or try/finally blocks to guarantee release even on exception paths. Verify by comparing checkout count against checkin count over time.

Aggregate demand exceeds downstream limit

The downstream connection limit is shared infrastructure. Options:

Reduce total workers across all app servers so aggregate pool demand fits the downstream limit.
Deploy a connection pooler (PgBouncer for PostgreSQL, ProxySQL for MySQL) to multiplex many application connections onto fewer downstream connections.
Reduce per-worker pool size and accept some checkout latency as a trade-off for staying within the downstream limit.

Prevention

Monitor downstream pool utilization alongside uWSGI metrics. The cascade is invisible in uWSGI metrics alone. The first sign is pool checkout wait time rising, which happens before busy ratio climbs. If your database client exposes pool wait metrics (SQLAlchemy does), alert on them.
Set application-level timeouts shorter than harakiri. Prevents workers from holding connections for the full harakiri window when the downstream is slow. Fail fast, return the connection, move on.
Size pools against downstream limits. Calculate total_app_servers x workers_per_server x pool_size_per_worker and compare against the downstream max_connections. Leave 20-30% headroom for other clients and administrative connections.
Use a connection pooler for shared downstreams. PgBouncer, ProxySQL, or equivalent multiplexers decouple your worker count from the database’s connection limit. This is the most effective structural fix.
Distinguish pool cascade from harakiri death spiral. In a harakiri death spiral, workers are killed and respawned at the timeout boundary, and throughput drops to near zero. In a pool cascade, workers are alive and blocked, and the busy ratio ramps gradually. Harakiri spirals need fail-fast logic or dependency restoration. Pool cascades need pool sizing or demand reduction.

How Netdata helps

Per-second worker busy ratio collection catches the gradual climb that distinguishes a cascade from a traffic spike. A cascade can tighten from 60% busy to 100% in under a minute, so 1-second granularity matters.
avg_rt per worker, collected at high frequency, shows the latency trend that precedes the busy ratio climb.
Exception rate tracking surfaces pool timeout errors at the WSGI layer as they begin.
Correlating uWSGI worker metrics (busy ratio, avg_rt, exceptions) with downstream database or cache metrics (connection counts, query latency) in a single dashboard makes the reinforcing loop visible at a glance.