$ guides / uwsgi / uwsgi-worker-exceptions ▌

Operations Guides

uWSGI worker exceptions climbing: unhandled errors reaching the WSGI layer

The workers[].exceptions counter is climbing, which means unhandled exceptions are propagating past your application code and reaching the uWSGI WSGI layer. Each increment typically corresponds to a 500 response delivered to the client. The counter is per-worker and monotonically increasing, so track the rate of change (delta over your polling interval), not the absolute value.

Critical nuance: this counter systematically undercounts application errors. If your framework (Django, Flask, FastAPI, and most others) catches exceptions via middleware and returns a 500 response itself, uWSGI never sees the exception. The request completes “successfully” from uWSGI’s perspective, and the counter does not increment. When the counter does climb, exceptions are escaping the framework entirely – a more severe condition than a framework-handled 500.

The diagnostic path depends on timing and scope. A simultaneous spike across all workers almost always indicates a downstream dependency failure, not a code bug. A spike that appears immediately after a deployment is almost certainly a code bug. A spike isolated to one worker points to corrupted per-worker state.

What this means

The exceptions counter lives on the worker slot struct in uWSGI’s shared memory. It is incremented when an exception propagates unhandled past the WSGI application callable. At that point, uWSGI catches it at the protocol layer, returns a 500 to the client, and increments the counter.

This counter is distinct from framework-level error handling. Consider the request flow:

flowchart TD
    A["Request arrives"] --> B["WSGI app callable invoked"]
    B --> C{"Exception raised?"}
    C -- No --> D["Normal response"]
    C -- Yes --> E{"Caught by framework middleware?"}
    E -- Yes --> F["Framework returns 500
exceptions NOT incremented"]
    E -- No --> G["uWSGI returns 500
exceptions INCREMENTED"]

The counter captures only path G. If your framework has a catch-all exception handler (most do), path E handles most errors silently from uWSGI’s perspective. This means:

Zero exceptions does not mean zero application errors. Your framework may be catching and returning 500s without uWSGI’s knowledge.
Non-zero exceptions means something is escaping the framework’s safety net entirely.
A rising rate is always worth investigating, even if absolute numbers look small relative to request volume.

Counter persistence: worker slot counters, including exceptions, are deliberately not reset on respawn (the uWSGI source comments say “do not reset worker counters on reload”); only delta_requests is reset. Always track the rate (delta between polling intervals), not the absolute value.

Common causes

Cause	What it looks like	First thing to check
Code bug after deploy	Exception spike starts within minutes of a deployment; may affect all workers or only those serving the new code path	Recent deployment history and changelog; application logs for traceback
Downstream dependency failure	Simultaneous spike across all workers; exceptions may be connection errors, timeouts, or unexpected None from failed calls	Database connectivity, external API health, cache availability
Resource exhaustion	Exceptions climb alongside OOM events or fd exhaustion; may present as MemoryError or OSError	dmesg for OOM kills; per-worker fd count vs ulimit
Corrupted worker state	Exceptions on one worker only while others are clean; may follow a specific pathological request	Per-worker exception breakdown; restart the affected worker
Deserialization or input errors	Exceptions correlate with specific endpoints or payload types; may be intermittent	Application logs for the specific error type; input validation

Quick checks

These commands are read-only and safe to run during an incident. Adjust the stats socket address (127.0.0.1:9191 in these examples) to match your deployment.

# Check total exceptions across all workers (point-in-time snapshot)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].exceptions] | add'

# Per-worker exception counts and request counts for ratio calculation
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.workers[] | select(.pid > 0) | {id: .id, exceptions: .exceptions, requests: .requests}'

# Compute the exception-to-request ratio across all workers
uwsgi --connect-and-read 127.0.0.1:9191 | jq '([.workers[].exceptions] | add) as $exc | ([.workers[].requests] | add) as $req | {exceptions: $exc, requests: $req, ratio_percent: (if $req > 0 then ($exc * 100 / $req) else 0 end)}'

# Check whether exceptions correlate with harakiri events
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].harakiri_count] | add'

<!-- Verified: the worker-level field is `respawn_count` (`.respawns` only appears for daemons/spoolers). -->
# Check worker respawn count (are workers crashing alongside exceptions?)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].respawn_count] | add'

# Check per-worker status and current URI (snapshot of what each worker is processing now)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.workers[] | select(.pid > 0) | {id: .id, status: .status, exceptions: .exceptions, uri: .uri}'

The .uri field shows the request currently being processed, not the request that caused the exception. It is only useful if you catch a worker mid-error.

For application-level error details, check the uWSGI log and your framework’s error log directly. The exceptions counter can climb without a corresponding traceback in uwsgi.log, which is a known pain point. uWSGI’s request log may not show the full traceback unless the framework itself logs the error or an exception-handler is configured.

How to diagnose it

Determine when the spike started. Pull exception counts at your normal polling interval and compute the delta. The first non-zero delta after a period of zero tells you when the problem began. Cross-reference with deployment timestamps.
Check scope: all workers or one? Use the per-worker breakdown. Exceptions climbing across all workers simultaneously points to shared state: a downstream dependency, a shared resource, or a code path all workers exercise. Only one worker affected suggests corrupted per-worker state.
Correlate with deployment events. If the spike began within minutes of a deploy, the new code is the likely cause. Check the deployment diff for changes to error handling, new endpoints, or modified dependency calls. This is the highest-probability cause when timing aligns.
Check downstream dependencies. If there is no recent deploy, check the health of everything the application talks to: databases, caches, external APIs. A dependency returning unexpected values (null where an object was expected, a schema change, a timeout) can cause unhandled exceptions in code that does not defensively handle the failure mode.
Look for the specific exception type in application logs. Even though uWSGI may not log the traceback, your framework or application might. Search for error-level log entries around the time the counter started climbing. If you use exception-handler or an external error tracker such as Sentry, check there.
Check for resource pressure. If exceptions correlate with memory pressure or fd exhaustion, the root cause is resource exhaustion, not a code bug. Check dmesg for OOM events and per-worker fd counts against ulimit -n.
Verify the exception-to-request ratio. Compute exceptions / requests over a recent window. A stable ratio means a consistent error rate. A suddenly increasing ratio means a new or worsening condition. Compare against your historical baseline.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Exception rate (delta of `workers[].exceptions`)	Primary signal: unhandled exceptions reaching WSGI layer	Any sustained non-zero rate in a previously zero-exception deployment
Exception-to-request ratio	Normalizes for traffic volume; more stable than absolute counts	Ratio exceeding historical baseline by 2x or more
Per-worker exception distribution	Distinguishes systemic issues (all workers) from local issues (one worker)	One worker accumulating exceptions while others stay clean
Harakiri count	Unhandled exceptions in C extensions may corrupt state and cause subsequent hangs (cautionary judgment; no documented causal link)	Harakiri rate rising alongside exception rate
Respawn count	Workers may crash due to exceptions, causing respawns	Respawn rate exceeding expected max-requests cadence
Request throughput	Rising exceptions may correlate with dropping throughput	Throughput declining while exception rate climbs
avg_rt	Exceptions on slow endpoints may correlate with latency spikes	avg_rt increasing alongside exception rate

Fixes

Code bug after deploy

Roll back the deployment. This is the fastest, most reliable fix. If rollback is not possible, identify the specific code path from application logs or error tracking, and deploy a hotfix that either fixes the bug or adds proper exception handling so the error is caught by the framework (returning a controlled 500 instead of an unhandled one).

Verify the fix by watching the exception rate drop to zero after the rollback or hotfix.

Downstream dependency failure

Fix the dependency, or add fail-fast handling in the application so that dependency failures produce caught exceptions (handled by the framework) rather than unhandled ones. For example, if a database query timeout raises an exception that escapes your error middleware, add explicit try/except around the database call with a controlled error response.

If the dependency is genuinely down, consider returning a 503 (service unavailable) at the load balancer level to prevent requests from reaching workers at all.

Corrupted worker state

If only one worker shows rising exceptions, that worker may have entered a bad state from a specific pathological request. Identify the worker ID and PID from the stats server, then recycle that worker. With max-requests configured, you can wait for natural recycling. To force it sooner, trigger a full uWSGI reload from the master process.

There is no per-worker reload signal in uWSGI: recycle individual workers via max-requests / reload-on-rss, or touch only that vassal’s config file in Emperor mode; a full master reload reloads all workers.

After recycling, verify the exception rate drops to zero on that worker. If it resumes immediately, the problem is not transient state but a code path triggered by specific input.

Resource exhaustion

Address the resource limit directly. For memory: check for leaks, tune max-requests or reload-on-rss. For file descriptors: raise ulimit -n and the systemd LimitNOFILE. See the file descriptor limits guide for details.

Using uWSGI exception handling options

uWSGI provides several options that affect how exceptions are handled after they occur. These are documented in the official uWSGI options reference:

catch-exceptions: Reports the exception traceback as HTTP output to the client. Discouraged for production use because it exposes internal details (it was broken with Python 3 until uWSGI 2.0.15, which fixed catch-exceptions with Python 3; a later Python 3.5+ segfault was fixed in 2.0.27).
reload-on-exception: Reloads the worker when any unhandled exception occurs. Useful for recovering from corrupted C-extension state, but causes capacity dips under sustained error conditions.
reload-on-exception-type: Reloads the worker only for a specific exception type. More targeted than blanket reload-on-exception.
exception-handler: Registers a custom exception handler. Can be stacked and supports plugins like Sentry. The exception-handler path uses a separate thread and is non-blocking (verified: the option predates 2.0.10, so there is no such minimum version).
backtrace-depth: Controls the depth of backtrace information for diagnostics.

These options do not prevent exceptions. They change what happens after one occurs. The underlying fix is always in the application code or the downstream dependency.

Prevention

Monitor the ratio, not the count. The exception-to-request ratio (exceptions / requests) normalizes for traffic volume. Alert on ratio exceeding historical baseline, not on a fixed absolute threshold.

Track rates, not absolute values. The counter may not reset on worker respawn. Always compute deltas between polling intervals. Alerting on absolute values leads to false positives after long-running workers accumulate legitimate exceptions over time.

Validate deploys against exception rates. Include the exception rate as a deployment gate. If the rate spikes immediately after a deploy, automated rollback or alerting should trigger before user impact scales.

Use error tracking integration. The exception-handler directive with a Sentry plugin (or equivalent) captures unhandled exceptions with full context before uWSGI returns the 500. This fills the gap between the counter (which tells you something happened) and the application log (which may not contain the traceback).

Do not rely on catch-exceptions in production. It leaks traceback information to clients. It was broken with Python 3 until uWSGI 2.0.15.

Distinguish uWSGI exceptions from framework 500s. Your framework’s error rate and uWSGI’s exception counter measure different things. If your framework returns 500s without incrementing the uWSGI counter, monitor both signals independently. The uWSGI counter catches only the most severe cases where error handling itself has failed.

How Netdata helps

Per-second exception rate detection. Netdata collects workers[].exceptions at high frequency, so you see the rate change within seconds of the first unhandled exception.
Exception-to-request ratio correlation. By collecting both exceptions and requests per worker, Netdata surfaces the ratio trend alongside absolute counts, making it easy to distinguish a real error spike from a traffic-driven increase.
Cross-worker distribution. Netdata shows per-worker exception counts, so you can immediately see whether the problem is systemic (all workers) or localized (one worker).
Harakiri and respawn correlation. When exceptions correlate with harakiri kills or worker respawns, Netdata’s unified timeline makes the causal chain visible without manual log correlation.
Deployment overlay. Netdata annotations let you mark deployment events on the timeline, making the deploy-to-exception-spike correlation immediate.