uWSGI behind nginx: 502 Bad Gateway and 'upstream prematurely closed connection'

The error string upstream prematurely closed connection while reading response header from upstream in the nginx error log means nginx had an established connection to a uWSGI worker, the worker accepted the request, and then the connection closed before nginx received a complete response header. The worker vanished mid-response.

This is distinct from two other errors that also produce 5xx responses but have different root causes:

This article covers the first row only. The root cause is always on the uWSGI side. The worker was killed by harakiri, crashed with a segfault, was killed by the kernel OOM killer, or was SIGKILL’d by the master enforcing an evil-reload-on-rss threshold. Diagnose from uWSGI, not from nginx.

The exit is always involuntary. Normal worker recycling via max-requests or reload-on-rss waits for the current request to finish before exiting. A worker that vanishes mid-response was killed by SIGKILL (from harakiri, from evil-reload-on-rss, or from the kernel OOM killer) or crashed with a segfault.

flowchart TD
    A["nginx returns 502"] --> B{"Error string in log?"}
    B -- "prematurely closed" --> C["Worker died mid-response"]
    B -- "Connection refused 111" --> D["Worker never accepted"]
    B -- "timed out 110" --> E["Worker too slow: 504"]
    C --> F{"harakiri_count rising?"}
    F -- "Yes" --> G["Request exceeded timeout"]
    F -- "No" --> H{"Non-harakiri respawns?"}
    H -- "Yes" --> I{"dmesg evidence?"}
    I -- "oom-kill" --> J["OOM killer"]
    I -- "segfault" --> K["C extension crash"]
    I -- "nothing" --> L["evil-reload-on-rss"]
    H -- "No" --> M["buffer-size overflow"]

Common causes

Quick checks

These commands are read-only and safe to run during an incident. Adjust the stats socket address (127.0.0.1:9191 in these examples) to match your deployment. If the stats server uses a UNIX socket, substitute uwsgi --connect-and-read /path/to/stats.sock. If --stats-http is enabled, curl http://127.0.0.1:9191 also works.

# Check harakiri count (per-worker, monotonic, never reset even on respawn)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].harakiri_count] | add'

# Check total respawn count (includes harakiri, max-requests, and crash respawns)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].respawn_count] | add'

# Check write errors (broken pipes from mid-request kills)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].cores[].write_errors] | add'

# Check per-worker RSS in MB (identify OOM or evil-reload-on-rss candidates)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.workers[] | select(.pid > 0) | {id: .id, rss_mb: (.rss / 1048576)}'

# Check currently busy workers and what they are serving
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.status == "busy") | .cores[] | select(.in_request == 1) | .vars[] | select(startswith("REQUEST_URI=")) | .[12:]]'

# Check for OOM kills in kernel log
dmesg -T | grep -i "out of memory\|oom-kill"

# Check for segfaults in kernel log
dmesg -T | grep -i "segfault"

# Confirm the exact nginx error string
grep "prematurely closed" /var/log/nginx/error.log | tail -20

How to diagnose it

1. Confirm the error type. Grep the nginx error log for the exact string. If it says Connection refused, the worker never accepted the connection: check socket permissions, whether uWSGI is listening, and whether the backlog is full. If it says timed out, the worker was too slow and nginx gave up first: that is a 504 latency problem, not a mid-response death.

2. Check harakiri_count delta. If it is rising, workers are being killed by the harakiri timer. The request exceeded the configured timeout. The root cause is a slow or hung request, typically a downstream dependency (database, external API, DNS resolution). Check the REQUEST_URI of busy workers in the stats server (inside each busy core’s vars array; there is no top-level uri field) to identify which endpoint is timing out. If harakiri-verbose is enabled, the uWSGI log will include the blocked syscall and wchan from /proc/<pid>/syscall and /proc/<pid>/wchan.

3. Isolate non-harakiri respawns. Subtract the harakiri_count delta from the respawn_count delta. Both are per-worker, monotonic counters that persist through respawns. The remainder includes max-requests recycling, segfaults, and OOM kills. If the non-harakiri respawn rate exceeds what max-requests recycling alone would produce, workers are crashing or being OOM-killed. Check dmesg for evidence.

4. Check for evil-reload-on-rss. If your config includes evil-reload-on-rss, the master sends SIGKILL to workers mid-request when RSS exceeds the threshold. This produces 502s on whatever request was in flight at that moment, with no harakiri increment. Check whether write_errors are elevated (clients disconnected before the response was sent). The fix is to switch to reload-on-rss, which is graceful: the worker finishes the current request, then exits and respawns.

5. Check buffer-size. If the 502s correlate with specific requests that have large headers (long URIs, large cookies), check the uWSGI log for invalid request block size. The default buffer-size is 4096 bytes. Requests with headers exceeding this are discarded, and nginx sees a closed connection. Note: this does not kill the worker. The worker stays alive, but the connection is closed without a response.

6. Align timeouts. Verify the relationship between nginx uwsgi_read_timeout and uWSGI’s harakiri. If uwsgi_read_timeout is shorter than harakiri, nginx returns a 504 to the client while the worker keeps processing (wasted capacity, and the worker eventually dies to harakiri anyway). If uwsgi_read_timeout is equal to or longer than harakiri, harakiri fires first, the worker dies, and nginx sees the disconnect as a 502. Set uwsgi_read_timeout >= harakiri so harakiri is the effective limiter.

Metrics and signals to monitor

Fixes

Harakiri kills: fix the downstream dependency

If harakiri_count is rising, the problem is not the timeout value. The problem is that requests are taking too long. Raising harakiri only delays the 502; it does not fix the root cause.

Identify the slow endpoint from the REQUEST_URI of busy workers in the stats server (per-core vars array). Check downstream dependencies: database query latency, external API response times, DNS resolution times. Add application-level timeouts on all downstream calls so a hung dependency returns a fast error instead of blocking until harakiri fires. Enable harakiri-verbose for diagnostic backtraces showing where the worker was blocked.

evil-reload-on-rss: switch to reload-on-rss

evil-reload-on-rss sends SIGKILL to workers mid-request when RSS exceeds the threshold. Every request in flight at that moment gets a 502. Replace it with reload-on-rss, which triggers a graceful exit: the worker finishes the current request, then recycles.

The tradeoff: reload-on-rss allows RSS to exceed the threshold during the grace period. If memory pressure is severe enough that the extra headroom matters, the real fix is addressing the memory leak or increasing system memory.

Segfaults: identify the C extension

Signal 11 (SIGSEGV) in the uWSGI log means a C extension crashed. Common culprits include database drivers, XML and HTML parsers, and image processing libraries. If segfaults correlate with specific request types, identify the code path and the library involved.

Persistent, unexplained segfaults can also indicate memory corruption from a previous OOM event. Treat them as suspicious until you identify the faulting library.

OOM kills: add memory headroom or reduce workers

The Linux OOM killer targets the highest-RSS process. With multiple uWSGI workers, one gets killed first, is respawned by the master, grows back, and the cycle repeats. Check dmesg for confirmation.

Options: reduce worker count to lower aggregate memory demand, configure reload-on-rss (graceful) at a threshold below the OOM danger zone so workers recycle before the kernel intervenes, or add system memory. Verify that max-requests is set so workers recycle proactively before RSS grows dangerously.

buffer-size overflow: increase the buffer

If the uWSGI log shows invalid request block size: X (max 4096), increase buffer-size to accommodate the largest expected request headers. Values of 16384 or 32768 are common for applications with large cookies or long URIs.

Prevention

• Configure harakiri and harakiri-verbose. Without harakiri, stuck workers have no timeout and permanently consume a worker slot. Set it to 2-3x your expected maximum legitimate request duration. The default is disabled. Enable harakiri-verbose for the blocked syscall and wchan when harakiri fires.
• Prefer reload-on-rss over evil-reload-on-rss. The graceful variant finishes the current request before recycling. The evil variant kills mid-request and produces 502s.
• Align uwsgi_read_timeout with harakiri. Set uwsgi_read_timeout >= harakiri so harakiri is the effective limiter and nginx sees the 502 (diagnostic) rather than timing out first and returning a 504 (less diagnostic, and the worker keeps processing uselessly).
• Monitor harakiri_count and respawn_count as rate-of-change deltas, not absolute values. Both are per-worker, monotonic counters that never reset, even on respawn.
• Set buffer-size appropriately for your workload if requests carry large headers.
• Ensure all downstream calls have their own timeouts. A missing downstream timeout on a database query, HTTP client, or DNS resolver is the most common root cause of harakiri storms and the 502s they produce.

How Netdata helps

• Per-second collection of harakiri_count, respawn_count, worker status, avg_rt, and RSS from the uWSGI stats server. You see the kill the moment it happens, not when nginx 502 alerts fire minutes later.
• Correlate harakiri_count spikes with downstream metrics (database query latency, external API response time) to identify the slow dependency causing the timeout.
• Anomaly detection on respawn_count flags crash-induced respawns separately from routine max-requests recycling.
• RSS growth-rate trends surface memory pressure before OOM kills or evil-reload-on-rss triggers.
• Write error rate monitoring catches mid-request kills (broken pipes from SIGKILL) that otherwise appear only as intermittent 502s in nginx logs.

Related guides

• uWSGI all workers busy: reading the busy ratio before the queue fills
• uWSGI avg_rt is not a real average: why the latency number lies
• uWSGI cache subsystem: hit ratio drops and ‘full’ insert failures
• uWSGI capacity planning: the leading indicators before saturation
• uWSGI chain reload: cycling workers one at a time for zero-downtime deploys
• uWSGI cheaper subsystem: dynamic worker scaling and the false ‘missing workers’ alert
• uWSGI connection pool cascade: downstream latency that stalls every worker
• uWSGI connection refused: clients turned away when the backlog overflows
• uWSGI Emperor healthy but vassal dead: monitoring each instance independently
• uWSGI file descriptor limits: raising ulimit -n and systemd LimitNOFILE
• uWSGI in gevent/async mode: why worker busy ratio stops meaning anything
• uWSGI threaded mode and the GIL: why more threads don’t add CPU parallelism