<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>uWSGI Operations Guides on Netdata</title><link>https://www.netdata.cloud/guides/uwsgi/</link><description>Recent content in uWSGI Operations Guides on Netdata</description><generator>Hugo</generator><language>en-us</language><atom:link href="https://www.netdata.cloud/guides/uwsgi/index.xml" rel="self" type="application/rss+xml"/><item><title>How uWSGI actually works in production: a mental model for operators</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-how-it-works-in-production/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-how-it-works-in-production/</guid><description>&lt;p>uWSGI is a pre-fork application server. A single master process spawns a pool of worker processes, each running a full copy of your application. The master never serves requests. It manages worker lifecycle, enforces timeouts, and coordinates graceful reloads. Every request flows through the same path: a client connects to a socket, the kernel queues that connection in a listen backlog, and a worker calls &lt;code>accept()&lt;/code> to pull it out and process it synchronously.&lt;/p></description></item><item><title>uWSGI all workers busy: reading the busy ratio before the queue fills</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-all-workers-busy/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-all-workers-busy/</guid><description>&lt;p>When every uWSGI worker shows &lt;code>status: &amp;quot;busy&amp;quot;&lt;/code>, the next incoming request does not wait in a place you can see. It lands in the kernel socket backlog, which uWSGI cannot reliably measure on standard Linux. If that backlog fills, the kernel drops connections silently. No log entry, no error counter, no uWSGI-level signal. The application looks alive but stops serving real traffic.&lt;/p>
&lt;p>The worker busy ratio is the earliest internal indicator that you are approaching that cliff. It tells you what proportion of your alive, non-cheaped workers are currently processing requests. Reading it correctly requires understanding what &amp;ldquo;busy&amp;rdquo; actually means, what it does not mean, and where the visibility gap starts.&lt;/p></description></item><item><title>uWSGI avg_rt is not a real average: why the latency number lies</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-avg-rt-not-cumulative/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-avg-rt-not-cumulative/</guid><description>&lt;p>uWSGI exposes a per-worker field called &lt;code>avg_rt&lt;/code> in its stats server JSON. Most monitoring tools label it &amp;ldquo;average response time&amp;rdquo; and graph it as a latency indicator. It is not a cumulative or lifetime average.&lt;/p>
&lt;p>The field is updated with the formula &lt;code>(old_avg_rt + current_request_time) / 2&lt;/code>, an exponential moving average with a smoothing factor of 0.5. The most recent request contributes 50% of the displayed value. The request before that contributes 25%. By the seventh request back, the contribution is under 1%.&lt;/p></description></item><item><title>uWSGI behind nginx: 502 Bad Gateway and 'upstream prematurely closed connection'</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-nginx-502-bad-gateway/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-nginx-502-bad-gateway/</guid><description>&lt;p>The error string &lt;code>upstream prematurely closed connection while reading response header from upstream&lt;/code> in the nginx error log means nginx had an established connection to a uWSGI worker, the worker accepted the request, and then the connection closed before nginx received a complete response header. The worker vanished mid-response.&lt;/p>
&lt;p>This is distinct from two other errors that also produce 5xx responses but have different root causes:&lt;/p>
&lt;table>
 &lt;thead>
 &lt;tr>
 &lt;th>nginx log string&lt;/th>
 &lt;th>HTTP status&lt;/th>
 &lt;th>What happened&lt;/th>
 &lt;th>Where to look&lt;/th>
 &lt;/tr>
 &lt;/thead>
 &lt;tbody>
 &lt;tr>
 &lt;td>&lt;code>upstream prematurely closed connection while reading response header from upstream&lt;/code>&lt;/td>
 &lt;td>502&lt;/td>
 &lt;td>Worker accepted the request, then died mid-response&lt;/td>
 &lt;td>uWSGI stats: &lt;code>harakiri_count&lt;/code>, &lt;code>respawn_count&lt;/code>; &lt;code>dmesg&lt;/code>&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>&lt;code>connect() ... failed (111: Connection refused) while connecting to upstream&lt;/code>&lt;/td>
 &lt;td>502&lt;/td>
 &lt;td>No connection established: uWSGI not listening, socket permissions, or backlog full&lt;/td>
 &lt;td>Master alive? Socket perms? &lt;code>ss -ltn&lt;/code>&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>&lt;code>upstream timed out (110: Connection timed out) while reading response header from upstream&lt;/code>&lt;/td>
 &lt;td>504&lt;/td>
 &lt;td>Worker accepted but did not finish within nginx&amp;rsquo;s timeout&lt;/td>
 &lt;td>uWSGI stats: worker busy ratio, &lt;code>avg_rt&lt;/code>&lt;/td>
 &lt;/tr>
 &lt;/tbody>
&lt;/table>
&lt;p>This article covers the first row only. The root cause is always on the uWSGI side. The worker was killed by harakiri, crashed with a segfault, was killed by the kernel OOM killer, or was SIGKILL&amp;rsquo;d by the master enforcing an &lt;code>evil-reload-on-rss&lt;/code> threshold. Diagnose from uWSGI, not from nginx.&lt;/p></description></item><item><title>uWSGI behind nginx: 504 Gateway Timeout when uwsgi_read_timeout fires first</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-nginx-504-gateway-timeout/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-nginx-504-gateway-timeout/</guid><description>&lt;p>A 504 from nginx with a &lt;code>uwsgi_pass&lt;/code> upstream means nginx stopped waiting for uWSGI before the worker finished. The default &lt;code>uwsgi_read_timeout&lt;/code> is 60 seconds. If uWSGI has no &lt;code>harakiri&lt;/code> configured (the default), or if harakiri is set higher than &lt;code>uwsgi_read_timeout&lt;/code>, the worker keeps processing a request whose response will never be read. The client already has their 504. The worker burns CPU, holds database connections, and occupies a slot that could serve real traffic.&lt;/p></description></item><item><title>uWSGI cache subsystem: hit ratio drops and 'full' insert failures</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-cache-full/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-cache-full/</guid><description>&lt;p>Response times are creeping upward on cached endpoints. Application logs show nothing. Worker busy ratio is normal. The cause is likely silent: uWSGI cache misses are increasing, and each miss forces the worker to compute or fetch the response instead of serving from shared memory.&lt;/p>
&lt;p>This article covers uWSGI&amp;rsquo;s built-in cache subsystem (&lt;code>cache2&lt;/code>), not external caches. If &lt;code>caches[]&lt;/code> is absent from the stats server output, the uWSGI cache is not enabled and these diagnostics do not apply.&lt;/p></description></item><item><title>uWSGI capacity planning: the leading indicators before saturation</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-capacity-saturation-indicators/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-capacity-saturation-indicators/</guid><description>&lt;p>uWSGI does not degrade gracefully. Performance looks fine until a hard limit is reached, then the service drops requests, kills workers, or refuses connections. Capacity planning is about knowing where each cliff is, measuring how close you are, and adding capacity before the edge.&lt;/p>
&lt;p>Four resources saturate as cliffs: worker pool (concurrency), memory (per-worker RSS), socket backlog (kernel listen queue), and file descriptors. Each has a distinct failure mode, leading indicator, and measurement method. Some signals come from the uWSGI stats server JSON. Others require external tools because uWSGI&amp;rsquo;s internal measurement is broken or silent.&lt;/p></description></item><item><title>uWSGI chain reload: cycling workers one at a time for zero-downtime deploys</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-chain-reload-zero-downtime/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-chain-reload-zero-downtime/</guid><description>&lt;p>The default uWSGI graceful reload sends all workers the shutdown signal at once. Each worker finishes its current request, exits, and the master forks a replacement. During the gap between old workers dying and new workers becoming ready to accept connections, serving capacity falls to zero. For applications with fast startup, this gap is a brief hiccup. For applications that load large models, warm connection pools, or run heavy imports on startup, the gap can stretch into seconds or minutes of complete unavailability.&lt;/p></description></item><item><title>uWSGI cheaper subsystem: dynamic worker scaling and the false 'missing workers' alert</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-cheaper-subsystem/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-cheaper-subsystem/</guid><description>&lt;p>The uWSGI cheaper subsystem dynamically scales the worker pool up and down at runtime based on demand. The master process spawns additional workers when traffic increases and reaps them when it subsides, reducing memory consumption during idle periods and providing automatic capacity during bursts.&lt;/p>
&lt;p>When the cheaper subsystem is active, the number of alive workers fluctuates by design. Cheaped workers (those scaled down by the subsystem) appear in the stats server output with &lt;code>&amp;quot;status&amp;quot;:&amp;quot;cheap&amp;quot;&lt;/code> and &lt;code>&amp;quot;pid&amp;quot;:0&lt;/code>. Monitoring that expects a fixed worker count will fire false &amp;ldquo;missing workers&amp;rdquo; alerts every time the system scales down.&lt;/p></description></item><item><title>uWSGI connection pool cascade: downstream latency that stalls every worker</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-connection-pool-cascade/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-connection-pool-cascade/</guid><description>&lt;p>Workers climbing toward all-busy. Response times creeping up. Throughput falling. No traffic spike, no deploy, no code change. The pattern built over minutes, not seconds.&lt;/p>
&lt;p>The mechanism is a reinforcing feedback loop. A downstream dependency (database, cache, external API) gets slower. Not dead, just slower. Each request now holds its downstream connection longer. The pool fills. The next request that needs a connection blocks waiting for one to be returned. That wait adds to the request&amp;rsquo;s wall-clock duration, which keeps the worker busy longer, which keeps the connection occupied longer. The loop tightens until every worker is blocked on pool checkout and throughput collapses.&lt;/p></description></item><item><title>uWSGI connection refused: clients turned away when the backlog overflows</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-connection-refused/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-connection-refused/</guid><description>&lt;p>Clients connecting to your uWSGI application receive connection refused or TCP RST. Behind nginx, the error log shows &lt;code>connect() failed (111: Connection refused) while connecting to upstream&lt;/code>. The application process may still be running, the master PID may exist, and the stats server may respond, yet real traffic is being turned away.&lt;/p>
&lt;p>This symptom has two root causes that look identical to the client but require opposite fixes. Either the listen backlog has overflowed because workers are saturated and cannot call &lt;code>accept()&lt;/code> fast enough, or the listener itself is dead (master gone, wrong socket path, broken socket permissions). Resolving that diagnostic fork is the first task.&lt;/p></description></item><item><title>uWSGI dropped connections: reading TcpExtListenOverflows and ListenDrops</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-tcp-listen-overflows/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-tcp-listen-overflows/</guid><description>&lt;p>Users report intermittent &amp;ldquo;connection refused&amp;rdquo; or timeouts. Nginx logs show 502s. uWSGI logs show nothing: no errors, no exceptions, no harakiri events. The master is alive, workers are accepting requests, throughput looks normal on average. But clients are being turned away.&lt;/p>
&lt;p>The explanation is in two kernel counters that uWSGI cannot report on its own: &lt;code>TcpExtListenOverflows&lt;/code> and &lt;code>TcpExtListenDrops&lt;/code>. When all uWSGI workers are busy and the kernel&amp;rsquo;s accept queue (the listen backlog) fills, the kernel drops new connections before uWSGI&amp;rsquo;s &lt;code>accept()&lt;/code> call ever runs. The application has no visibility into this event. The only evidence lives in &lt;code>/proc/net/netstat&lt;/code>.&lt;/p></description></item><item><title>uWSGI Emperor healthy but vassal dead: monitoring each instance independently</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-emperor-vassal-down/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-emperor-vassal-down/</guid><description>&lt;p>The Emperor process is running, its stats endpoint responds, and it continues scanning config directories. But one application is down. Clients get connection refused or timeouts. Your monitoring says the Emperor is healthy because it is. Emperor liveness is not application liveness.&lt;/p>
&lt;p>In Emperor mode, the Emperor manages one vassal (a full uWSGI instance) per config file. Each vassal is an independent process tree with its own master, workers, and stats server. The Emperor monitors config files and spawns or reloads vassals, but a vassal can die and fail to restart on a broken config while the Emperor runs unaffected. Vassal states:&lt;/p></description></item><item><title>uWSGI file descriptor limits: raising ulimit -n and systemd LimitNOFILE</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-file-descriptor-limits/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-file-descriptor-limits/</guid><description>&lt;p>The default per-process file descriptor limit on many Linux distributions is 1024. For a uWSGI instance running multiple workers, each holding connections, sockets, log files, and database handles, that ceiling is too low for production.&lt;/p>
&lt;p>File descriptor exhaustion in uWSGI is silent. When the limit is hit, &lt;code>accept()&lt;/code> and &lt;code>open()&lt;/code> calls fail with &lt;code>EMFILE&lt;/code>. New connections are rejected with no uWSGI-level error, no log entry, and no stats counter reflecting the problem. Clients see connection resets or timeouts. The master process stays alive. Worker status looks normal. The only evidence is at the OS level.&lt;/p></description></item><item><title>uWSGI harakiri death spiral: workers killed and respawned while throughput collapses</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-harakiri-death-spiral/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-harakiri-death-spiral/</guid><description>&lt;p>Workers are being killed by harakiri and respawned in a tight loop. Every request blocks past the configured timeout. The master sends SIGKILL to the worker, forks a replacement, and the new worker immediately accepts the next queued request, which also blocks. Throughput collapses to near zero while the worker pool appears &amp;ldquo;busy&amp;rdquo; at or near 100%.&lt;/p>
&lt;p>This is the harakiri death spiral: a composite failure where the root cause is almost never uWSGI itself. A downstream dependency (database, external API, DNS resolver) has become unresponsive or unreachable. Every request that touches that dependency hangs. Harakiri is working as designed, killing stuck workers, but the recycling provides no relief because the next request is equally doomed.&lt;/p></description></item><item><title>uWSGI harakiri not configured: stuck workers with no timeout and no recovery</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-harakiri-not-configured/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-harakiri-not-configured/</guid><description>&lt;p>uWSGI workers are vanishing one at a time. The master process is alive, the stats server responds, and &lt;code>harakiri_count&lt;/code> reads zero across every worker. By every metric you thought mattered, the service looks healthy. Then you notice that half the workers have been in &lt;code>busy&lt;/code> status for the last ten minutes without processing a single new request. Requests are timing out at the proxy. The listen queue is filling. The application is effectively down, and nothing in uWSGI is attempting to recover it.&lt;/p></description></item><item><title>uWSGI HARAKIRI ON WORKER: requests killed for exceeding the timeout</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-harakiri-worker-killed/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-harakiri-worker-killed/</guid><description>&lt;pre tabindex="0">&lt;code>HARAKIRI ON WORKER N (pid: XXXX, try: 1) !!!
&lt;/code>&lt;/pre>&lt;p>You see it in the uWSGI error log, followed by the master reporting the worker died by signal 9. Your monitoring shows a spike in 502 or 504 responses from the reverse proxy. Something in the request path is hanging long enough to exceed the configured &lt;code>harakiri&lt;/code> limit, and the master process is killing workers to prevent total pool exhaustion.&lt;/p>
&lt;p>Harakiri is doing its job. It is not the problem. The problem is whatever is causing requests to run past the timeout.&lt;/p></description></item><item><title>uWSGI harakiri timeout: setting it against request duration and nginx timeouts</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-harakiri-timeout-tuning/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-harakiri-timeout-tuning/</guid><description>&lt;p>The harakiri timeout is uWSGI&amp;rsquo;s per-request watchdog: if a request runs longer than the configured threshold, the master kills the worker (SIGKILL by default) and respawns it. Without it, a hung worker stays hung indefinitely, consuming a slot in the pool until every worker is stuck and the service is dark.&lt;/p>
&lt;p>Harakiri has to sit in a narrow band: above the p99 of legitimate requests so you do not kill real traffic, but below the point where the upstream proxy gives up. If nginx&amp;rsquo;s &lt;code>uwsgi_read_timeout&lt;/code> fires first, you get a 504 while the uWSGI worker keeps processing a response nobody will read. If harakiri fires first, nginx sees an upstream disconnect and returns a 502. The capacity implications differ: a 504 means wasted work, a 502 means a recycled worker.&lt;/p></description></item><item><title>uWSGI harakiri-verbose: finding the blocked syscall behind a timeout</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-harakiri-verbose-diagnosis/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-harakiri-verbose-diagnosis/</guid><description>&lt;p>When a uWSGI worker exceeds the harakiri timeout, the master kills it with SIGKILL and respawns a replacement. The default harakiri log line identifies which worker died and when, but not what the worker was doing when it got stuck. The request could be CPU-bound (a pathological regex, a tight loop), blocked on I/O (a database query that never returns), or deadlocked on an internal lock. Each requires a different fix.&lt;/p></description></item><item><title>uWSGI in gevent/async mode: why worker busy ratio stops meaning anything</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-gevent-async-monitoring/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-gevent-async-monitoring/</guid><description>&lt;p>Switching uWSGI from pre-fork to gevent mode lets a single worker multiplex dozens or hundreds of concurrent requests on an event loop. The tradeoff: the worker busy ratio, the primary capacity metric in pre-fork mode, becomes unreliable. A worker reports &amp;ldquo;busy&amp;rdquo; whenever its event loop is running, which is nearly always, regardless of actual request load.&lt;/p>
&lt;p>Teams that keep alerting on busy ratio after switching to gevent either see perpetual 100% utilization alarms or, worse, silence while real problems go undetected. The cheaper_busyness algorithm has a known incompatibility with gevent that causes workers to spawn under load but never scale back down.&lt;/p></description></item><item><title>uWSGI listen backlog and net.core.somaxconn: sizing the connection queue</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-listen-backlog-somaxconn/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-listen-backlog-somaxconn/</guid><description>&lt;p>The connection backlog is the buffer between arriving TCP connections and your uWSGI workers. It is set by two independent values that must agree: uWSGI&amp;rsquo;s &lt;code>--listen&lt;/code> option and the kernel&amp;rsquo;s &lt;code>net.core.somaxconn&lt;/code>. The effective backlog is the smaller of the two. If either is too small, the queue fills during brief traffic spikes or downstream slowdowns, and the kernel starts dropping connections silently.&lt;/p>
&lt;p>The defaults are both low. uWSGI ships with &lt;code>--listen 100&lt;/code>. Linux kernels before 5.4 default &lt;code>net.core.somaxconn&lt;/code> to 128; kernels 5.4 and later raised the default to 4096. On any service handling hundreds of requests per second, a 200-millisecond downstream hiccup fills a 100-slot queue in under a second. The result is intermittent client-side connection errors (timeouts or &amp;ldquo;connection refused&amp;rdquo;) with nothing in the uWSGI logs.&lt;/p></description></item><item><title>uWSGI listen queue full: the backlog overflow that drops connections silently</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-listen-queue-full/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-listen-queue-full/</guid><description>&lt;p>Clients report connection timeouts or resets. Your load balancer shows 502s or 504s on requests to the uWSGI backend. The uWSGI master is running, workers are alive, the stats server responds, and application logs show no errors. This is listen queue overflow: the kernel accept backlog on the uWSGI listening socket is full, and the kernel is silently dropping new connections before &lt;code>accept()&lt;/code>.&lt;/p>
&lt;p>The listen queue is the kernel-level buffer between completed TCP handshakes and available workers. When all workers are busy and this buffer fills, the kernel silently drops incoming connections. No uWSGI log entry is written. No uWSGI error counter increments. The stats server runs in the master process and stays responsive during complete worker starvation, so health checks that query the stats endpoint still pass. Health check endpoints handled by workers may also pass if they are lightweight enough to be accepted between drops. Meanwhile, real users cannot connect.&lt;/p></description></item><item><title>uWSGI listen_queue always zero: why the stats field is broken on Linux</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-listen-queue-stats-unreliable/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-listen-queue-stats-unreliable/</guid><description>&lt;p>You open the uWSGI stats server JSON during a traffic spike. &lt;code>listen_queue&lt;/code> reads &lt;code>0&lt;/code>. &lt;code>load&lt;/code> reads &lt;code>0&lt;/code>. &lt;code>listen_queue_errors&lt;/code> reads &lt;code>0&lt;/code>. But nginx is returning 502s, clients are seeing connection refused, and all workers are busy.&lt;/p>
&lt;p>The fields do not measure what you think. On standard Linux, &lt;code>listen_queue&lt;/code> is broken. &lt;code>load&lt;/code> is identical to &lt;code>listen_queue&lt;/code> (the source code has a &lt;code>TODO&lt;/code> comment admitting this). &lt;code>listen_queue_errors&lt;/code> is dead code that is never incremented. All three read &lt;code>0&lt;/code> regardless of actual socket backlog pressure.&lt;/p></description></item><item><title>uWSGI master process dead: total outage while the PID file lingers</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-master-process-dead/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-master-process-dead/</guid><description>&lt;p>Nginx returns 502. The uWSGI PID file exists at its expected path, so monitoring reports the service as &amp;ldquo;up.&amp;rdquo; But nothing is serving traffic. The master process is dead, and the stale PID file is lying to you.&lt;/p>
&lt;p>The master holds the listening socket, forks workers, enforces harakiri timeouts, handles graceful reloads, and serves the stats endpoint. When it dies, workers are gone. No connections are accepted. The outage is total.&lt;/p></description></item><item><title>uWSGI max-requests: worker recycling that masks leaks and crashes</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-max-requests-recycling/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-max-requests-recycling/</guid><description>&lt;p>You set &lt;code>max-requests&lt;/code> because it bounds memory leaks in uWSGI workers. Workers recycle on schedule. But max-requests does not fix leaks. It bounds them by killing the worker before the leak becomes fatal.&lt;/p>
&lt;p>Two blind spots follow. First, peak RSS per worker is &lt;code>leak_rate x max_requests&lt;/code>, not zero. At 1MB leaked per request and &lt;code>max-requests 1000&lt;/code>, each worker reaches roughly 1GB before recycling. With 8 workers, that is 8GB consumed by leak tolerance alone. Second, the respawn activity from max-requests looks identical to crash churn in the stats server. If workers are also dying from segfaults, OOM kills, or harakiri timeouts, the respawn counter alone cannot tell you why.&lt;/p></description></item><item><title>uWSGI monitoring checklist: the signals every production app server needs</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-monitoring-checklist/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-monitoring-checklist/</guid><description>&lt;p>The stats server (enabled with &lt;code>--stats &amp;lt;socket&amp;gt;&lt;/code>) exports a JSON document with worker state, request counters, memory usage, and error rates. This checklist organizes those signals by maturity level, from survival to expert. Each level assumes the previous one is in place.&lt;/p>
&lt;p>uWSGI is in maintenance mode (bugfixes only, no new features), so the stats schema and signal semantics are stable. Three fields are not: &lt;code>listen_queue&lt;/code>, &lt;code>load&lt;/code>, and &lt;code>listen_queue_errors&lt;/code> are broken or dead code on standard Linux. This checklist flags every unreliable field and points to the external measurement that works instead.&lt;/p></description></item><item><title>uWSGI monitoring maturity model: from survival to expert</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-monitoring-maturity-model/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-monitoring-maturity-model/</guid><description>&lt;p>A four-level progression for uWSGI monitoring, from bare liveness checks to deep signal correlation. The levels are cumulative: you cannot skip to Level 3 by tracking RSS growth trends while ignoring harakiri rate. Each tier closes a specific class of blind spot that the previous tier could not see.&lt;/p>
&lt;p>The model assumes the uWSGI stats server is enabled with &lt;code>--stats &amp;lt;address&amp;gt;&lt;/code>. Without it, every level above survival is unreachable. HTTP access to the stats server requires the additional &lt;code>--stats-http&lt;/code> flag; otherwise use &lt;code>uwsgi --connect-and-read &amp;lt;addr&amp;gt;&lt;/code> for TCP sockets or &lt;code>socat - UNIX-CONNECT:&amp;lt;path&amp;gt;&lt;/code> for UNIX sockets.&lt;/p></description></item><item><title>uWSGI processes vs threads: sizing concurrency without wasting memory</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-processes-vs-threads/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-processes-vs-threads/</guid><description>&lt;p>The concurrency model you choose in uWSGI sets how many requests your server handles simultaneously, what that costs in memory, and what your monitoring signals mean once traffic arrives. Get it wrong and you either waste memory on idle processes or starve the kernel listen queue because you misread what &amp;ldquo;all workers busy&amp;rdquo; indicates.&lt;/p>
&lt;p>uWSGI supports three concurrency models. Pre-fork gives you one request per worker process. Threaded gives you N requests per worker via threads. Async (gevent or asyncio) gives you many requests per worker via an event loop. Each has a different memory profile, a different CPU characteristic, and a different definition of &amp;ldquo;busy.&amp;rdquo;&lt;/p></description></item><item><title>uWSGI reload blackout: a broken deploy leaves zero workers running</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-reload-blackout/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-reload-blackout/</guid><description>&lt;p>You deployed new code to uWSGI. The graceful reload killed the old workers, but throughput is zero. The master process is alive. Every worker it forks dies before accepting a connection.&lt;/p>
&lt;p>The reload mechanism worked. The new code cannot initialize. Workers crash during startup, the master respawns them, and they crash again. No worker stays alive long enough to serve a request.&lt;/p>
&lt;p>The master process appears healthy. Process checks pass. The stats server responds to polling. If your health check only verifies the master PID or pings the stats endpoint, it reports green while 100% of requests are dropped at the kernel backlog.&lt;/p></description></item><item><title>uWSGI reload thundering herd: capacity drops to zero during a slow restart</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-graceful-reload-thundering-herd/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-graceful-reload-thundering-herd/</guid><description>&lt;p>You deploy a new version of your application. Seconds later, request throughput collapses to zero. Nginx returns 502s. The uWSGI master process is alive and the stats server responds, but no workers are accepting connections. Thirty seconds to several minutes later, throughput recovers. If this pattern aligns exactly with every deployment, you are hitting the reload thundering herd.&lt;/p>
&lt;p>A standard uWSGI graceful reload kills all workers simultaneously, then forks replacements. If your application has a slow startup phase (heavy imports, ML model loading, cache warming, database connection pool initialization), there is a window where zero workers are ready to serve. The kernel listen queue fills with pending connections, overflows, and drops new connections silently. The first requests that land after workers recover hit cold caches and empty connection pools, making them abnormally slow.&lt;/p></description></item><item><title>uWSGI reload-on-rss vs evil-reload-on-rss: recycling workers on memory limits</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-reload-on-rss/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-reload-on-rss/</guid><description>&lt;p>Two uWSGI options recycle workers when their resident set size crosses a threshold: &lt;code>reload-on-rss&lt;/code> and &lt;code>evil-reload-on-rss&lt;/code>. Both produce the same effect on a monitoring dashboard. The &lt;code>respawn_count&lt;/code> counter increments, RSS drops to baseline, a new worker is forked. The memory sawtooth looks healthy. From the master process perspective, the two are indistinguishable.&lt;/p>
&lt;p>The difference is in what happens to the in-flight request when the kill occurs.&lt;/p>
&lt;p>&lt;code>reload-on-rss&lt;/code> is graceful. The master signals the worker to shut down. The worker finishes its current request, delivers the full response, and exits. The replacement worker picks up the next connection from the listen queue. No client sees anything unusual.&lt;/p></description></item><item><title>uWSGI respawn rate high: telling crashes apart from max-requests recycling</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-respawn-rate-high/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-respawn-rate-high/</guid><description>&lt;p>You see &lt;code>respawn_count&lt;/code> climbing across your uWSGI workers. The question is whether workers are crashing or whether uWSGI is doing what you configured it to do.&lt;/p>
&lt;p>&lt;code>respawn_count&lt;/code> is a single monotonic counter per worker slot that increments for every reason a worker can die and come back: &lt;code>max-requests&lt;/code> recycling, &lt;code>reload-on-rss&lt;/code> memory recycling, harakiri timeout kills, segfaults, OOM kills, and manual &lt;code>kill -9&lt;/code>. The raw number tells you workers are churning. It does not tell you why.&lt;/p></description></item><item><title>uWSGI response time climbing: rising avg_rt and where it comes from</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-response-time-high/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-response-time-high/</guid><description>&lt;p>When avg_rt climbs in the uWSGI stats server, the instinct is to look for slow application code. That is often right, but avg_rt is more nuanced than a simple average response time. Understanding what it actually measures is the difference between a fast diagnosis and a misleading rabbit hole.&lt;/p>
&lt;p>avg_rt is an exponential moving average (EMA), not a cumulative average. When it climbs, something per-request is getting slower. The question is whether the cause is a downstream dependency (database pool exhaustion, slow external API), resource contention (CPU saturation, GC storms, GIL pressure in threaded Python), or cascading starvation (workers backing up because each request holds the worker longer). Correlating avg_rt with the worker busy ratio narrows this quickly.&lt;/p></description></item><item><title>uWSGI RSS vs VSZ: reading worker memory without being fooled by shared pages</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-rss-vs-vsz/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-rss-vs-vsz/</guid><description>&lt;p>You look at your uWSGI stats and see 8 workers each reporting 500 MB of RSS. The host has 4 GB of RAM. By the math, you should be deep into swap, but &lt;code>vmstat&lt;/code> shows zero swap activity and response times are fine.&lt;/p>
&lt;p>The gap is copy-on-write (COW) sharing. uWSGI&amp;rsquo;s default model loads the application once in the master process, then forks workers that inherit the master&amp;rsquo;s memory. Linux marks those inherited pages as shared and read-only. As long as workers do not write to them, the pages stay shared across all workers. But Linux RSS counts every shared page fully for every process that maps it. Sum worker RSS across 8 workers and you may overstate real memory consumption by 2-5x.&lt;/p></description></item><item><title>uWSGI spooler backlog: deferred tasks piling up on disk</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-spooler-backlog/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-spooler-backlog/</guid><description>&lt;p>The uWSGI spooler is an on-disk deferred-job queue. Application code serializes work units into files in a spool directory, and dedicated spooler processes consume them asynchronously. When producers enqueue work faster than the spooler drains it, or when the spooler crashes or stalls, files accumulate on disk. The backlog grows silently, deferred work latency increases, and in the worst case the disk fills or tasks are silently dropped.&lt;/p>
&lt;p>A spooler backlog does not affect user-facing HTTP latency. Workers continue serving traffic. The degradation is in the deferred pipeline: emails arrive late, reports are stale, notifications lag. This makes the backlog easy to miss until background jobs stop running entirely.&lt;/p></description></item><item><title>uWSGI stats server unreachable: socket permissions and a stuck master</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-stats-server-unreachable/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-stats-server-unreachable/</guid><description>&lt;p>Your monitoring collector reports the uWSGI stats server as unreachable. Before debugging the stats server, answer one question: is uWSGI actually serving traffic?&lt;/p>
&lt;p>The uWSGI stats server runs inside the master process and serves raw JSON over a socket (UNIX or TCP) when enabled with &lt;code>--stats&lt;/code>. The master does not serve application requests. This separation matters: the stats server can be unreachable while every worker is processing traffic, and it can be reachable while every worker is stuck. Stats reachability and application availability are independent signals.&lt;/p></description></item><item><title>uWSGI stats server: enabling it and reading it without --stats-http</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-stats-server-setup/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-stats-server-setup/</guid><description>&lt;p>The uWSGI stats server is the primary data source for every monitoring signal in the uWSGI playbook: worker busy ratio, harakiri count, avg_rt, respawn rate, RSS, exception counts, and more. Without it, you are blind to internal state. With it, you have complete visibility into worker pool health, saturation, and failure modes.&lt;/p>
&lt;p>The stats server must be explicitly enabled with the &lt;code>--stats&lt;/code> option. It is not on by default. Once enabled, it serves a JSON document containing the full internal state of the master process and all workers, including PIDs, worker status, request counters, memory usage, and per-core request data.&lt;/p></description></item><item><title>uWSGI stats socket exposure: internal state leaking on a public port</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-stats-socket-exposure/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-stats-socket-exposure/</guid><description>&lt;p>The uWSGI stats server, enabled with &lt;code>--stats&lt;/code>, serves a JSON blob containing the complete internal state of the master and all workers: PIDs, memory usage, request counts, response times, configuration details, and in-flight request data. When this socket is reachable from an untrusted network, every field in that JSON is an information-disclosure vector.&lt;/p>
&lt;p>The stats server has no authentication. Access control is entirely network-level. Any client that can connect to the socket address receives the full dump with no challenge, no token, no ACL. The common pattern &lt;code>--stats :9191&lt;/code> (address with no IP specified) binds on every interface, which means the stats endpoint is open to anyone who can reach the host on that port.&lt;/p></description></item><item><title>uWSGI threaded mode and the GIL: why more threads don't add CPU parallelism</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-gil-cpu-bound/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-gil-cpu-bound/</guid><description>&lt;p>You doubled the thread count on each uWSGI worker, expecting throughput to scale with your multi-core box. CPU utilization barely moved. Response time did not improve. Memory looks fine. The queue still fills under load.&lt;/p>
&lt;p>The cause is almost certainly the CPython Global Interpreter Lock. In threaded mode, a uWSGI worker runs N OS threads, but all N threads share a single GIL within that process. For CPU-bound Python code, only one thread executes bytecode at a time regardless of how many threads you configured. Adding threads helps with I/O-bound workloads, where threads can yield the GIL while waiting on network or disk, but it does nothing for computation-heavy request handlers.&lt;/p></description></item><item><title>uWSGI throughput drop: requests per second falling with traffic steady</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-throughput-drop/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-throughput-drop/</guid><description>&lt;p>Throughput in uWSGI is a derived metric: sum per-worker &lt;code>requests&lt;/code> counters across all workers, then compute the delta between polling intervals. Anything that disrupts those counters or changes how fast workers complete requests shows up as a throughput trend.&lt;/p>
&lt;p>Two caveats before you start:&lt;/p>
&lt;ol>
&lt;li>
&lt;p>&lt;strong>Throughput can rise during failure.&lt;/strong> If the app starts returning 500s immediately without processing, requests-per-second goes up while useful throughput goes down. Always pair throughput with exception rate.&lt;/p></description></item><item><title>uWSGI thundering herd: accept() contention and the thunder-lock fix</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-thundering-herd-thunder-lock/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-thundering-herd-thunder-lock/</guid><description>&lt;p>You have a uWSGI deployment with multiple worker processes, and something does not add up. CPU usage is elevated. Workers toggle between idle and busy. But request throughput is low, response times are higher than expected, and adding more workers makes things worse instead of better. The system looks under load but is not actually doing much work.&lt;/p>
&lt;p>This is the uWSGI thundering herd problem. When a new connection arrives on the shared listening socket, every idle worker process wakes up and races to call &lt;code>accept()&lt;/code>. Only one wins. The rest burn CPU and kernel time on a context switch for nothing, then go back to sleep. Under low-to-moderate traffic with many workers, this contention consumes more CPU than the actual request processing.&lt;/p></description></item><item><title>uWSGI too many open files: file descriptor exhaustion and EMFILE</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-too-many-open-files/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-too-many-open-files/</guid><description>&lt;p>When the uWSGI process hits its file descriptor limit (&lt;code>RLIMIT_NOFILE&lt;/code>), &lt;code>accept()&lt;/code> and &lt;code>open()&lt;/code> start returning &lt;code>EMFILE&lt;/code> (errno 24). New connections are silently rejected. Logging fails. Application code throws exceptions that look like disk or network failures. The uWSGI stats endpoint does not report file descriptor usage, so there is no uWSGI-level signal pointing to the real problem. Operators typically chase disk space, I/O, or network connectivity before realizing the process has simply run out of file descriptors.&lt;/p></description></item><item><title>uWSGI worker exceptions climbing: unhandled errors reaching the WSGI layer</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-worker-exceptions/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-worker-exceptions/</guid><description>&lt;p>The &lt;code>workers[].exceptions&lt;/code> counter is climbing, which means unhandled exceptions are propagating past your application code and reaching the uWSGI WSGI layer. Each increment typically corresponds to a 500 response delivered to the client. The counter is per-worker and monotonically increasing, so track the rate of change (delta over your polling interval), not the absolute value.&lt;/p>
&lt;p>Critical nuance: this counter systematically undercounts application errors. If your framework (Django, Flask, FastAPI, and most others) catches exceptions via middleware and returns a 500 response itself, uWSGI never sees the exception. The request completes &amp;ldquo;successfully&amp;rdquo; from uWSGI&amp;rsquo;s perspective, and the counter does not increment. When the counter does climb, exceptions are escaping the framework entirely &amp;ndash; a more severe condition than a framework-handled 500.&lt;/p></description></item><item><title>uWSGI worker killed by the OOM killer: mysterious respawns under memory pressure</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-worker-oom-killed/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-worker-oom-killed/</guid><description>&lt;p>You see worker respawns in the uWSGI log that do not match your &lt;code>max-requests&lt;/code> cadence. The master reports workers dying from signal 9 (&lt;code>SIGKILL&lt;/code>), and either harakiri is not configured or the harakiri count is zero. Workers come back, serve traffic for a while, then die again. The interval shrinks over time.&lt;/p>
&lt;p>This is the signature of the Linux OOM killer targeting uWSGI workers. As total worker RSS grows beyond available RAM, the kernel swaps, performance degrades, and the OOM killer selects the largest process on the system. In a uWSGI deployment, that process is almost always a worker. The master respawns it, the new worker re-imports the application, RSS climbs back, and the cycle repeats.&lt;/p></description></item><item><title>uWSGI worker memory leak: RSS climbing across all workers</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-worker-memory-leak/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-worker-memory-leak/</guid><description>&lt;p>Per-worker RSS climbs steadily. All workers track each other in lockstep. After a restart, RSS looks stable for hours or days, then the pattern repeats. With &lt;code>max-requests&lt;/code> or &lt;code>reload-on-rss&lt;/code> configured, you see a sawtooth: RSS rises, a worker recycles, RSS drops, then rises again. Without those directives, workers eventually hit the OOM killer or start swapping.&lt;/p>
&lt;p>The cause may be a genuine leak in application code, Python allocator fragmentation that never returns pages to the OS, a C extension bug, or Docker file-descriptor inflation. The first two are operationally identical: RSS grows monotonically and does not come back down without a process recycle.&lt;/p></description></item><item><title>uWSGI worker pool starvation: the silent outage where every worker is busy</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-worker-pool-starvation/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-worker-pool-starvation/</guid><description>&lt;p>Every worker shows &lt;code>status: &amp;quot;busy&amp;quot;&lt;/code>. The master process is alive. The stats server responds instantly. Your load balancer health check returns 200. But real users are seeing timeouts, connection refused errors, or hanging pages. This is worker pool starvation.&lt;/p>
&lt;p>The mechanism is a concurrency cliff. uWSGI&amp;rsquo;s pre-fork model assigns one request per worker at a time in the default configuration. When every worker is occupied with a slow or hanging request, new connections pile into the kernel listen queue (socket backlog). Once that queue fills, the kernel silently drops connections with no uWSGI log entry, no error counter, and no exception. The service is dead for real traffic while every surface-level health signal stays green.&lt;/p></description></item><item><title>uWSGI worker respawn loop: 'DAMN ! worker died :( trying respawn'</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-worker-respawn-loop/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-worker-respawn-loop/</guid><description>&lt;p>The uWSGI master logs &lt;code>DAMN ! worker N (pid: XXX) died, killed by signal S :( trying respawn ...&lt;/code> when it receives SIGCHLD for a worker that exited unexpectedly. The next line is typically &lt;code>Respawned uWSGI worker N (new pid: XXXX)&lt;/code>. In a respawn loop, these pairs repeat rapidly: the master forks a replacement, the replacement dies, and the cycle continues.&lt;/p>
&lt;p>The signal number &lt;code>S&lt;/code> is your primary diagnostic clue. Signal 9 (SIGKILL) means OOM kill or harakiri. Signal 11 (SIGSEGV) means a C extension crashed. Signal 6 (SIGABRT) means an assert failed. A worker that dies less than one second after spawn is hitting a startup failure, not a request-time problem.&lt;/p></description></item><item><title>uWSGI worker segfault: SIGSEGV in a C extension and the respawn that follows</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-worker-segfault/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-worker-segfault/</guid><description>&lt;p>A worker dies with signal 11. The master respawns it. The uWSGI log records &lt;code>DAMN ! worker N (pid: XXXX) died, killed by signal 11 :( trying respawn ...&lt;/code> followed by &lt;code>Respawned uWSGI worker N (new pid: YYYY)&lt;/code>. From the outside, the service looks like it survived a momentary blip. It did not. A SIGSEGV means something in C-level code crashed: a compiled extension (numpy, lxml, a database driver, OpenSSL, protobuf), a uWSGI internal bug, or memory corruption. Python application code cannot normally produce SIGSEGV.&lt;/p></description></item><item><title>uWSGI worker stuck in busy: a hung request that never returns</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-stuck-worker/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-stuck-worker/</guid><description>&lt;p>One worker sits in &lt;code>busy&lt;/code> far longer than any legitimate request should take. The others cycle through &lt;code>idle&lt;/code> and &lt;code>busy&lt;/code> normally. The stuck worker is not crashing, not erroring, and not completing. It is holding a request open indefinitely and will stay that way until something kills it.&lt;/p>
&lt;p>This is the single-worker poisoning pattern. A specific request triggered a pathological code path: blocking I/O without a timeout, a regex catastrophe, an unbounded database query, or a deadlock in a C extension. The worker called &lt;code>accept()&lt;/code>, began processing, and never returned. Its request counter is frozen. Its &lt;code>running_time&lt;/code> stopped advancing. One slot of your worker pool is permanently consumed.&lt;/p></description></item><item><title>uWSGI write and read errors: broken pipes and clients that disconnect</title><link>https://www.netdata.cloud/guides/uwsgi/uwsgi-write-read-errors/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/uwsgi/uwsgi-write-read-errors/</guid><description>&lt;p>write_errors and read_errors are per-core counters in the uWSGI stats server. write_errors increments when a socket write fails during response delivery, almost always because the client disconnected before the response finished (broken pipe). read_errors increments when the connection is lost during request body reading. Some of both are normal: users navigate away, mobile clients switch networks, browsers cancel pending requests. A sustained rise in write_errors, especially when correlated with rising response time or worker respawns, points to a real problem.&lt;/p></description></item></channel></rss>