uWSGI spooler backlog: deferred tasks piling up on disk

The uWSGI spooler is an on-disk deferred-job queue. Application code serializes work units into files in a spool directory, and dedicated spooler processes consume them asynchronously. When producers enqueue work faster than the spooler drains it, or when the spooler crashes or stalls, files accumulate on disk. The backlog grows silently, deferred work latency increases, and in the worst case the disk fills or tasks are silently dropped.

A spooler backlog does not affect user-facing HTTP latency. Workers continue serving traffic. The degradation is in the deferred pipeline: emails arrive late, reports are stale, notifications lag. This makes the backlog easy to miss until background jobs stop running entirely.

Spooler stats appear in the spoolers[] array of the uWSGI stats server JSON, but only when spoolers are configured. Key fields: tasks (pending count), running (0 or 1), and respawns. Growing tasks means consumption is falling behind production. A rising oldest-task age (measured from the filesystem) means the spooler is stuck or processing very slowly. Non-zero respawns means the spooler process is crashing and being restarted by the master.

What this means

The question is always: why is the consumer too slow, or why is the producer too fast? The failure breaks down into four categories:

1. The spooler process is dead or crashing. The master respawns it, but if each new spooler crashes immediately (import error, permissions issue, segfault in a C extension), no tasks are consumed. Files accumulate. The respawns counter climbs while tasks never decreases.

2. The spooler is alive but tasks fail and requeue. The task handler raises an exception, and the spooler return value causes the task to be retried rather than removed. The file stays. On the next poll cycle, the spooler picks it up, fails again, and repeats. Poison tasks block the queue.

3. The spooler is alive but slow. Task handlers take too long (slow database queries, external API calls without timeouts). The spooler processes tasks sequentially and cannot keep up with the enqueue rate.

4. The spooler is alive but not picking up new tasks. The spooler has no filesystem-event notification: it polls the spool directory on a fixed cycle (default every 30 seconds, --spooler-frequency). If the spooler is stuck in a long-running task, crashed, or its poll cycle is not running, newly written files sit untouched until a cycle that actually scans them.

flowchart TD
    A["tasks growing"] --> B{"running field?"}
    B -->|0 or fluctuating| C["Spooler crashing"]
    B -->|1| D{"respawns climbing?"}
    D -->|yes| C
    D -->|no| E{"oldest task age?"}
    E -->|very old, stuck| F["Poison task or stuck handler"]
    E -->|moderate, growing| G["Spooler too slow"]
    C --> H["Check error log"]
    F --> I["Check app log for exceptions"]
    G --> J["Check handler duration"]
    A --> K{"disk nearly full?"}
    K -->|yes| L["Disk full blocking writes"]
    K -->|no| B

Common causes

Quick checks

# Check spooler stats from the stats server
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.spoolers[]'

# Count pending files in the spool directory
ls -1 /var/spool/uwsgi/ 2>/dev/null | wc -l

# Check age of oldest unprocessed task
ls -1rt /var/spool/uwsgi/ 2>/dev/null | head -1 | xargs -I{} stat --format='%Y' /var/spool/uwsgi/{}

# Check if the spooler process is alive
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.spoolers[] | {running, respawns, tasks}'

# Check disk space on the spool filesystem
df -h /var/spool/uwsgi/

# Check the configured spooler poll interval (default 30 seconds)
grep -E "spooler-frequency|spooler-freq" /etc/uwsgi/app.ini

# Check spool directory ownership and permissions
ls -ld /var/spool/uwsgi/

The stats server address (127.0.0.1:9191 in these examples) and the spool directory path (/var/spool/uwsgi/) vary by deployment. If your stats server uses a UNIX socket, replace --connect-and-read 127.0.0.1:9191 with --connect-and-read /path/to/stats.sock. If --stats-http is enabled, curl http://127.0.0.1:9191 also works.

How to diagnose it

Step 1: Confirm the backlog exists and is growing

Pull the spooler stats twice with a known interval between polls:

uwsgi --connect-and-read 127.0.0.1:9191 | jq '.spoolers[] | {tasks, running, respawns}'
sleep 30
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.spoolers[] | {tasks, running, respawns}'

If tasks increased between polls, the backlog is growing. If running is 0, the spooler process is not alive. If respawns increased, the spooler is crashing.

Step 2: Check if the spooler process is alive

The running field tells you whether the master considers the spooler active. If running is 0 and respawns is climbing, the spooler is in a crash loop. Check the uWSGI error log for the crash cause:

grep -i "spooler" /var/log/uwsgi/app.log | tail -50

Common crash causes include:

• Application import errors in the spooler handler module
• Permission denied on the spool directory. The master can create directories as root before dropping privileges via --uid, leaving them owned by root and inaccessible to the unprivileged spooler process (uWSGI issue #1858; fixed by commit e368cf5).
• Segfaults in C extensions called by the task handler

Step 3: Check the filesystem directly

The stats server reports tasks as a count, but the filesystem tells you more. List the spool directory by modification time:

# List oldest files first
ls -1rt /var/spool/uwsgi/ | head -20

# Count total files
ls -1 /var/spool/uwsgi/ | wc -l

# Check the oldest file's age (epoch timestamp and name)
ls -1rt /var/spool/uwsgi/ | head -1 | xargs -I{} stat --format='%Y %n' /var/spool/uwsgi/{}

If the oldest file is hours or days old, the spooler is either stuck or processing so slowly it will never catch up. If the directory has thousands of files, ext4 directory scanning performance may be degrading throughput.

Step 4: Look for poison tasks

A poison task causes the handler to raise an exception every time it runs. The spooler function’s return value determines what happens to the file:

• SPOOL_OK (-2): The task file is removed. Success.
• SPOOL_RETRY (-1): The task is retried on the next poll cycle. The file stays.
• SPOOL_IGNORE (0): the task is treated as not handled by the plugin: the file is left in place and is picked up again on a later poll cycle.

Any other return value — including no return / None in Python — is treated as a retry: the file is left in place and retried on a later cycle.

If your handler raises an unhandled exception, the spooler may treat it as a retry, leaving the file in place. The same task blocks the queue, consuming a processing slot on every poll cycle.

Check application logs for recurring exceptions:

grep -i "spooler\|task" /var/log/uwsgi/app.log | grep -i "error\|exception\|traceback" | tail -30

Step 5: Check disk space and the spooler poll cycle

# Disk space on the spool filesystem
df -h /var/spool/uwsgi/

# Filesystem free inodes (task files are small; inode exhaustion is possible)
df -i /var/spool/uwsgi/

# Confirmed files present but untouched: compare directory mtime with spooler poll times
ls -l --time-style=full-iso /var/spool/uwsgi/ | head

If the disk is full, new task files cannot be written and producers will fail silently or loudly depending on error handling. The spooler itself never uses inotify: it discovers tasks only when a poll cycle scans the directory, so a filled or pathologically slow filesystem can delay discovery beyond the configured --spooler-frequency interval.

Metrics and signals to monitor

Fixes

Spooler process is crashing

Check the uWSGI error log for the crash cause.

Permission denied on spool directory. Pre-create the spool directory with correct ownership before starting uWSGI. The master can create directories as root before dropping privileges, leaving them owned by root and inaccessible to the unprivileged spooler process.

Import error in task handler. Verify that the spooler handler module imports cleanly in the uWSGI context. Import failures crash the process on each poll cycle.

Segfault in C extension. If the spooler crashes with SIGSEGV (signal 11), a C extension called by the task handler is likely responsible. Check for memory corruption in C extension code invoked from the handler.

Tasks raising and requeuing

Poison tasks block the queue because the spooler retries them indefinitely. Options:

Fix the handler. The root cause is an exception in the task processing code. Identify it from application logs and fix the bug.

Move poison tasks aside. Manually move the offending files out of the spool directory to unblock the queue. This is a temporary fix while the handler bug is being addressed.

# CAUTION: Stop the spooler first, or accept the race with active spooler file operations.
# Moving files from under a running spooler can cause claim conflicts or missed tasks.
mkdir -p /var/spool/uwsgi-quarantine
find /var/spool/uwsgi/ -type f -mmin +60 -exec mv {} /var/spool/uwsgi-quarantine/ \;

Add a max-attempts mechanism. Track retry counts within the task payload or a sidecar store, and return SPOOL_OK (removing the file) after N retries to prevent infinite loops.

Spooler is slow

If task processing is inherently slow (heavy computation, slow external calls), increase parallelism or reduce per-task cost:

Multiple spooler processes. Use --spooler-processes <n> to run N spooler processes against the same spool directory. Each process independently scans and claims files, increasing throughput.

Reduce per-task cost. Profile the task handler and optimize the slowest operations. Add timeouts to external calls so a single slow dependency does not monopolize the spooler.

Adjust poll frequency. The default spooler poll frequency is 30 seconds, configurable via --spooler-frequency <secs> (alias --spooler-freq). If tasks are time-sensitive, reduce this value. More frequent polling increases filesystem load, which matters when thousands of files are present.

Use spooler-harakiri. --spooler-harakiri <secs> sets a timeout for spooler tasks. Without it, a single hung task blocks the spooler indefinitely, similar to how request workers hang without --harakiri.

Disk full

Clear old processed files (if your spooler leaves them behind) or move the spool directory to a larger filesystem. The spooler has no built-in “attempts” or failed-task subdirectory feature — failed (retried) tasks stay in the spool directory, so implement retry counting in the handler (see “Add a max-attempts mechanism”) to bound the disk they occupy.

Spooler not picking up new files

The spooler does not use inotify or any filesystem-event notification; it discovers tasks only when a poll cycle scans the spool directory (every --spooler-frequency seconds, default 30). If new files sit untouched while the spooler is alive:

# Confirm the spooler process is running and not stuck in a task
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.spoolers[] | {running, tasks, respawns}'

# Check the master log for a task that never completes
grep -i "spooler" /var/log/uwsgi/app.log | tail -50

A task that blocks forever (no spooler-harakiri) leaves the spooler unable to scan for new work; a crashed spooler is restarted by the master (respawns climbs) and rescans on its next cycle.

ext4 dir_index degradation

With thousands of files in a single spool directory, ext4 HTree directory indexing can degrade scan performance. The spooler must readdir the directory on every poll cycle, and large directories make this expensive.

Shard tasks across subdirectories. The spooler always writes task files into priority subdirectories named by the task’s priority. --spooler-ordered makes the spooler scan those numbered subdirectories in order (sorted scan), which distributes work across directories. Otherwise, consider sharding at the application level by writing tasks to date-prefixed or hash-prefixed subdirectories.

Clean up regularly. Ensure processed tasks are removed promptly.

Consider tmpfs. tmpfs eliminates disk I/O entirely but loses durability on restart. Use only if task loss on crash is acceptable.

Prevention

• Monitor spoolers[].tasks as a trend, not a threshold. A static count of 50 pending tasks might be normal. Sustained growth is the warning sign. Track the delta over time and alert on the slope.
• Monitor spoolers[].respawns. Any non-zero respawn rate for the spooler is abnormal. Unlike worker respawns (which include expected max-requests recycling), spooler respawns always indicate a crash.
• Alert on oldest task age. Even if tasks is stable, a high oldest-task age means deferred work is delayed beyond its expected SLA.
• Pre-create spool directories with correct ownership. Avoid the permission race where the master creates directories as root before dropping privileges.
• Add max-attempts to task handlers. Poison tasks that retry forever block the queue. Implement a retry limit and return SPOOL_OK after exhaustion.
• Watch disk usage on the spool filesystem. A filling disk will eventually prevent new task writes, causing silent task loss from the producer side.
• Run multiple spooler processes if task volume warrants it. A single spooler process is a serialization bottleneck by design. --spooler-processes removes it.
• Configure spooler-harakiri. Without a timeout, a single hung task blocks the spooler indefinitely. Set it to a reasonable multiple of your expected task duration.

How Netdata helps

Netdata collects the uWSGI stats server JSON and exposes spooler metrics alongside worker, memory, and request signals at per-second resolution. Useful correlations for spooler backlog:

• spoolers[].tasks trend at per-second resolution. A backlog that grows during traffic spikes and drains during lulls indicates a capacity issue. Monotonic growth with no drain indicates a crashed or stuck spooler. Per-second collection captures patterns that 30-second polling misses.
• spoolers[].respawns correlated with crash events. Spikes in spooler respawns, correlated with disk I/O metrics and system logs, help isolate the crash cause.
• Disk I/O and filesystem fullness. Disk metrics show I/O wait, read/write rates, and filesystem usage on the partition holding the spool directory. Correlating disk saturation with rising tasks distinguishes “slow disk” from “slow handler.”
• Worker pool health. If request workers are saturated, they may stop enqueuing tasks at the normal rate. This can mask a spooler backlog: tasks appears stable because production stopped, not because consumption improved.
• Anomaly detection on task count. A slowly growing backlog can look normal in absolute terms but anomalous relative to historical patterns.

Related guides

• uWSGI all workers busy: reading the busy ratio before the queue fills
• uWSGI avg_rt is not a real average: why the latency number lies
• uWSGI chain reload: cycling workers one at a time for zero-downtime deploys
• uWSGI cheaper subsystem: dynamic worker scaling and the false ‘missing workers’ alert
• uWSGI connection refused: clients turned away when the backlog overflows
• uWSGI in gevent/async mode: why worker busy ratio stops meaning anything
• uWSGI threaded mode and the GIL: why more threads don’t add CPU parallelism
• uWSGI reload thundering herd: capacity drops to zero during a slow restart
• uWSGI harakiri death spiral: workers killed and respawned while throughput collapses
• uWSGI harakiri not configured: stuck workers with no timeout and no recovery
• uWSGI harakiri timeout: setting it against request duration and nginx timeouts
• uWSGI harakiri-verbose: finding the blocked syscall behind a timeout