$ guides / uwsgi / uwsgi-harakiri-verbose-diagnosis ▌

Operations Guides

uWSGI harakiri-verbose: finding the blocked syscall behind a timeout

When a uWSGI worker exceeds the harakiri timeout, the master kills it with SIGKILL and respawns a replacement. The default harakiri log line identifies which worker died and when, but not what the worker was doing when it got stuck. The request could be CPU-bound (a pathological regex, a tight loop), blocked on I/O (a database query that never returns), or deadlocked on an internal lock. Each requires a different fix.

harakiri-verbose bridges that gap. When enabled, the master reads /proc/<pid>/syscall and /proc/<pid>/wchan for the worker at the moment of the kill and logs both. This turns “requests are slow” into “requests are blocked in recvfrom() on the database socket” or “workers are deadlocked on a futex.”

What harakiri-verbose captures

When the harakiri timer fires, the master reads two procfs files for the worker before sending SIGKILL:

/proc/<pid>/syscall: the syscall number the worker is blocked in, followed by its arguments and register values.
/proc/<pid>/wchan: the kernel wait channel, a string naming the kernel function where the process is sleeping.

The master formats these into two log lines:

HARAKIRI: -- syscall> <syscall_nr> <arg0> <arg1> ... <arg7>
HARAKIRI: -- wchan> <wait_channel_string>

The syscall number is a raw integer, architecture-dependent: syscall 7 is poll on x86_64 but a different number on ARM64. Decode it using ausyscall --dump, the /usr/include/asm/unistd_64.h header, or a syscall table for your architecture.

The wchan string is kernel-version-dependent but generally more immediately readable. futex_wait_queue_me means the worker is sleeping on a lock. 0 or running means the worker was on CPU, not blocked in a syscall.

This feature is Linux-only. The relevant code in core/master_utils.c is wrapped in #ifdef __linux__. On macOS or BSD, enabling the flag produces no additional output.

flowchart TD
    A["Harakiri fires on worker"] --> B["Master reads /proc/pid/syscall"]
    A --> C["Master reads /proc/pid/wchan"]
    B --> D["Log: syscall number + args"]
    C --> E["Log: wchan string"]
    D --> F["Decode syscall to name"]
    E --> F
    F --> G{"wchan is 0 or running?"}
    G -->|Yes| H["Worker was CPU-bound"]
    G -->|No| I["Correlate with URI from stats"]
    I --> J["Identify endpoint + dependency"]

Prerequisites

Harakiri must be configured. Without harakiri = <seconds>, the timer never fires and harakiri-verbose has nothing to log. See uWSGI harakiri not configured.
Linux kernel. The /proc/<pid>/syscall and /proc/<pid>/wchan interfaces are Linux-specific.
Log access. The output goes to the master’s log destination (stderr or the configured log file).
Stats server (recommended). While not required for harakiri-verbose itself, the stats server’s per-core vars dump includes the REQUEST_URI of in-flight requests, which lets you correlate the killed worker with the endpoint it was serving.

Enabling harakiri-verbose

Add the flag alongside your existing harakiri configuration:

[uwsgi]
harakiri = 30
harakiri-verbose = true

Or on the command line:

uwsgi --harakiri 30 --harakiri-verbose ...

The flag is boolean: present or absent. There is no harm in leaving it enabled permanently. The overhead is two procfs reads at the moment of a harakiri event, which is already an exceptional condition.

Reload uWSGI for the flag to take effect. You will only see output when the next harakiri event occurs, so verification is indirect: confirm the flag is in your loaded configuration, then wait for the next kill.

Reading the output

When harakiri fires, look for the two HARAKIRI: lines in the log.

Step 1: decode the syscall number

# Decode specific syscall numbers on x86_64
ausyscall --dump | grep -Ew '(7|42|45|47|202|232)'

Common syscalls seen in harakiri-verbose output on x86_64:

Syscall number	Name	What it means
202	futex	Blocked on a kernel futex (lock contention)
45	recvfrom	Blocked reading from a socket
47	recvmsg	Blocked reading a message from a socket
42	connect	Blocked establishing a network connection
7	poll	Waiting for I/O readiness
232	epoll_wait	Waiting for I/O readiness (epoll)

These numbers are x86_64-specific. On ARM64 or other architectures, the numbers differ but the syscall names are the same. Always decode against the architecture of the host where the worker was killed.

Step 2: read the wchan string

The wchan gives you the kernel function where the process is sleeping. Three categories cover most cases:

wchan value	Interpretation
`futex_wait_queue_me`	Worker is sleeping on a futex. Lock contention: `uwsgi.lock()`, a threading lock, or the Python GIL.
`0` or `running`	Worker was actively on CPU when the harakiri timer expired. CPU-bound work, not I/O.
Any other non-zero string	Worker is blocked in the kernel. The decoded syscall name tells you what kind of operation it was.

When wchan shows 0, the worker was not sleeping in any kernel function. It was executing user-space code: an infinite loop, a pathological regex, or heavy computation that never yields.

Step 3: correlate with the serving URI

The syscall and wchan tell you what the worker was doing. The stats server tells you what endpoint it was serving: the URI is inside each busy core’s vars array as a REQUEST_URI=... entry (the stats JSON has no top-level uri field). Together, they localize both the endpoint and the dependency.

# Check the URI of busy workers from the stats server
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.status == "busy") | .cores[] | select(.in_request == 1) | .vars[] | select(startswith("REQUEST_URI=")) | .[12:]]'

If all harakiri kills show recvfrom in the syscall log and the busy workers are serving /api/export, the export endpoint has a database query that hangs. If the kills show futex and the workers are on diverse endpoints, you have global lock contention, not a per-endpoint issue.

The vars data is only present while a core is processing a request, and only if --stats-no-cores is not set. It is a point-in-time snapshot. If the worker was already killed by the time you poll, the request may be gone. In that case, use the harakiri-verbose log line timestamp and correlate it with access logs for the same worker around the same time.

Common pitfalls

wchan> 0 from a procfs read failure. If the worker exits or is killed by something else between the harakiri decision and the procfs read, the master cannot read /proc/<pid>/wchan and logs 0. This is a race condition, not a diagnostic signal. If you see wchan> 0 alongside syscall> running, the procfs read returned no useful data. Look at the next harakiri event for actionable output.

Threaded mode kills the entire process. In multi-threaded workers, harakiri sends SIGKILL to the worker process, not to the specific thread that was stuck. One slow thread causes all threads in that worker to die. The harakiri-verbose log reflects the state of the process as a whole, which may not identify the offending thread. If you see harakiri kills on threaded workers with a wchan of futex_wait_queue_me, one thread may be holding a lock that blocks the others.

uwsgi.lock() deadlock after harakiri. If a worker holding uwsgi.lock() is killed by harakiri, the lock is never released. All subsequent workers that try to acquire the same lock block on futex_wait_queue_me and eventually get killed by harakiri themselves. The harakiri-verbose output for the waiting workers correctly shows the futex, but the root cause is the original killed worker that did not release the lock. A cascade of futex-blocked harakiri kills across multiple workers points to an unreleased lock from a prior kill, not independent contention.

Architecture-dependent syscall numbers. The syscall number is a raw integer read from /proc/<pid>/syscall. On x86_64, syscall 7 is poll. On x86 (32-bit), the number is different. On ARM64, yet another number. If you operate across architectures, decode the number for each host individually. Do not assume a single mapping applies fleet-wide.

Harakiri firing during after-request hooks. If you have after-request hooks that take significant time, harakiri can fire during the hook rather than during request processing. The harakiri-verbose output then reflects what the hook was doing, not the request itself. Use harakiri-no-arh = true to disable the harakiri timer during after-request hooks if this distorts your diagnosis.

Signals to monitor alongside harakiri-verbose

Harakiri-verbose gives you the root cause of individual kills. These signals give you the pattern across the fleet:

Signal	Why it matters	Warning sign
Harakiri rate (delta of `harakiri_count`)	Sustained non-zero rate means requests are systematically hanging, not just occasionally slow	Any rate above zero in a deployment where harakiri is normally zero
Worker busy ratio	When harakiri fires and busy ratio is near 100%, the pool is saturated and each kill reduces capacity further	Sustained 80%+ with harakiri events
Average response time (`avg_rt`)	When `avg_rt` approaches the harakiri timeout, more kills are imminent	`avg_rt` rising toward the configured harakiri value
Respawn rate	Every harakiri kill equals one respawn. If respawn rate tracks harakiri rate 1:1, respawns are harakiri-caused, not `max-requests` recycling	Respawn rate significantly above the expected max-requests cadence

For the full harakiri death spiral pattern (harakiri rate rising, throughput collapsing, workers churning), see uWSGI harakiri death spiral.

How Netdata helps

Netdata’s per-second metrics complement the per-kill root cause from harakiri-verbose:

Harakiri count per worker (delta): a worker with a rising kill count while others stay stable points to a request-specific code path.
Worker busy ratio: shown alongside harakiri events on the same timeline, confirming whether the pool is saturated at the moment kills fire.
Per-worker avg_rt: per-second granularity shows latency approaching the harakiri threshold before the first kill.
Respawn rate vs harakiri rate: correlating these two deltas distinguishes harakiri-driven respawns from routine max-requests recycling without manual subtraction.
Anomaly detection: anomaly flags on harakiri rate, response time, and worker busy ratio surface deviations from baseline before they reach static alert thresholds.