$ guides / uwsgi / uwsgi-spooler-backlog ▌

Operations Guides

uWSGI spooler backlog: deferred tasks piling up on disk

The uWSGI spooler is an on-disk deferred-job queue. Application code serializes work units into files in a spool directory, and dedicated spooler processes consume them asynchronously. When producers enqueue work faster than the spooler drains it, or when the spooler crashes or stalls, files accumulate on disk. The backlog grows silently, deferred work latency increases, and in the worst case the disk fills or tasks are silently dropped.

A spooler backlog does not affect user-facing HTTP latency. Workers continue serving traffic. The degradation is in the deferred pipeline: emails arrive late, reports are stale, notifications lag. This makes the backlog easy to miss until background jobs stop running entirely.

Spooler stats appear in the spoolers[] array of the uWSGI stats server JSON, but only when spoolers are configured. Key fields: tasks (pending count), running (0 or 1), and respawns. Growing tasks means consumption is falling behind production. A rising oldest-task age (measured from the filesystem) means the spooler is stuck or processing very slowly. Non-zero respawns means the spooler process is crashing and being restarted by the master.

What this means

The question is always: why is the consumer too slow, or why is the producer too fast? The failure breaks down into four categories:

The spooler process is dead or crashing. The master respawns it, but if each new spooler crashes immediately (import error, permissions issue, segfault in a C extension), no tasks are consumed. Files accumulate. The respawns counter climbs while tasks never decreases.
The spooler is alive but tasks fail and requeue. The task handler raises an exception, and the spooler return value causes the task to be retried rather than removed. The file stays. On the next poll cycle, the spooler picks it up, fails again, and repeats. Poison tasks block the queue.
The spooler is alive but slow. Task handlers take too long (slow database queries, external API calls without timeouts). The spooler processes tasks sequentially and cannot keep up with the enqueue rate.
The spooler is alive but cannot see new tasks. Filesystem or inotify limits prevent the spooler from detecting newly written files. The spooler polls an empty-looking directory while producers fill it.

flowchart TD
    A["tasks growing"] --> B{"running field?"}
    B -->|0 or fluctuating| C["Spooler crashing"]
    B -->|1| D{"respawns climbing?"}
    D -->|yes| C
    D -->|no| E{"oldest task age?"}
    E -->|very old, stuck| F["Poison task or stuck handler"]
    E -->|moderate, growing| G["Spooler too slow"]
    C --> H["Check error log"]
    F --> I["Check app log for exceptions"]
    G --> J["Check handler duration"]
    A --> K{"disk nearly full?"}
    K -->|yes| L["Disk full blocking writes"]
    K -->|no| B

Common causes

Cause	What it looks like	First thing to check
Spooler process crashing	`respawns` climbing, `running` fluctuating between 0 and 1, `tasks` growing monotonically	uWSGI error log for spooler crash messages or tracebacks
Tasks raising and requeuing	`tasks` count stable or slowly growing, same files reappear in spool dir, no respawns	Application error logs for recurring exceptions from the task handler
Disk full or nearly full	New task writes fail or are rejected, `tasks` stops growing but existing tasks are not processed	`df -h` on the filesystem containing the spool directory
inotify watch limits	Spooler does not pick up new files, old tasks processed but new ones sit untouched	`cat /proc/sys/fs/inotify/max_user_watches` and system-level inotify usage
Slow task processing	`tasks` grows during traffic spikes, drains slowly during lulls, oldest task age correlates with task count	Application logs for slow operations within the spooler handler
ext4 dir_index degradation	Spooler appears alive but task processing is extremely slow, thousands of files in one directory	File count in spool directory, filesystem type and mount options
Spooler directory permissions	Spooler stuck in respawn loop with Permission denied errors	Ownership and permissions of the spool directory vs the configured uid/gid

Quick checks

# Check spooler stats from the stats server
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.spoolers[]'

# Count pending files in the spool directory
ls -1 /var/spool/uwsgi/ 2>/dev/null | wc -l

# Check age of oldest unprocessed task
ls -1rt /var/spool/uwsgi/ 2>/dev/null | head -1 | xargs -I{} stat --format='%Y' /var/spool/uwsgi/{}

# Check if the spooler process is alive
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.spoolers[] | {running, respawns, tasks}'

# Check disk space on the spool filesystem
df -h /var/spool/uwsgi/

# Check inotify watch limits
cat /proc/sys/fs/inotify/max_user_watches

# Check spool directory ownership and permissions
ls -ld /var/spool/uwsgi/

The stats server address (127.0.0.1:9191 in these examples) and the spool directory path (/var/spool/uwsgi/) vary by deployment. If your stats server uses a UNIX socket, replace --connect-and-read 127.0.0.1:9191 with --connect-and-read /path/to/stats.sock. If --stats-http is enabled, curl http://127.0.0.1:9191 also works.

How to diagnose it

Step 1: Confirm the backlog exists and is growing

Pull the spooler stats twice with a known interval between polls:

uwsgi --connect-and-read 127.0.0.1:9191 | jq '.spoolers[] | {tasks, running, respawns}'
sleep 30
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.spoolers[] | {tasks, running, respawns}'

If tasks increased between polls, the backlog is growing. If running is 0, the spooler process is not alive. If respawns increased, the spooler is crashing.

Step 2: Check if the spooler process is alive

The running field tells you whether the master considers the spooler active. If running is 0 and respawns is climbing, the spooler is in a crash loop. Check the uWSGI error log for the crash cause:

grep -i "spooler" /var/log/uwsgi/app.log | tail -50

Common crash causes include:

Application import errors in the spooler handler module
Permission denied on the spool directory. The master can create directories as root before dropping privileges via --uid, leaving them owned by root and inaccessible to the unprivileged spooler process.
Segfaults in C extensions called by the task handler

Step 3: Check the filesystem directly

The stats server reports tasks as a count, but the filesystem tells you more. List the spool directory by modification time:

# List oldest files first
ls -1rt /var/spool/uwsgi/ | head -20

# Count total files
ls -1 /var/spool/uwsgi/ | wc -l

# Check the oldest file's age (epoch timestamp and name)
ls -1rt /var/spool/uwsgi/ | head -1 | xargs -I{} stat --format='%Y %n' /var/spool/uwsgi/{}

If the oldest file is hours or days old, the spooler is either stuck or processing so slowly it will never catch up. If the directory has thousands of files, ext4 directory scanning performance may be degrading throughput.

Step 4: Look for poison tasks

A poison task causes the handler to raise an exception every time it runs. The spooler function’s return value determines what happens to the file:

SPOOL_OK (-2): The task file is removed. Success.
SPOOL_RETRY (-1): The task is retried on the next poll cycle. The file stays.
SPOOL_IGNORE (0): The task is skipped.

If your handler raises an unhandled exception, the spooler may treat it as a retry, leaving the file in place. The same task blocks the queue, consuming a processing slot on every poll cycle.

Check application logs for recurring exceptions:

grep -i "spooler\|task" /var/log/uwsgi/app.log | grep -i "error\|exception\|traceback" | tail -30

Step 5: Check disk space and inotify limits

# Disk space on the spool filesystem
df -h /var/spool/uwsgi/

# inotify watch limits
cat /proc/sys/fs/inotify/max_user_watches

# Current inotify instance limits
cat /proc/sys/fs/inotify/max_user_instances

If the disk is full, new task files cannot be written and producers will fail silently or loudly depending on error handling. If inotify watches are exhausted, the spooler may not receive filesystem events for new files.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`spoolers[].tasks`	Pending task count. Direct measure of backlog depth.	Sustained growth over multiple polling intervals
`spoolers[].running`	Whether the spooler process is alive.	Drops to 0 or fluctuates between 0 and 1
`spoolers[].respawns`	How many times the spooler has been restarted by the master.	Any non-zero rate (delta over time)
Oldest task age (filesystem)	How long the oldest pending file has been waiting.	Age exceeds your SLA for deferred work
Spool directory file count	Total files on disk. Correlates with filesystem scan performance.	Thousands of files may degrade ext4 scan speed
Disk usage on spool filesystem	Whether the disk is filling with spool files.	Approaching 90% or higher
Worker busy ratio	Whether request workers (which enqueue tasks) are saturated.	If request workers are stuck, they may stop producing tasks, masking the backlog

Fixes

Spooler process is crashing

Check the uWSGI error log for the crash cause.

Permission denied on spool directory. Pre-create the spool directory with correct ownership before starting uWSGI. The master can create directories as root before dropping privileges, leaving them owned by root and inaccessible to the unprivileged spooler process.

Import error in task handler. Verify that the spooler handler module imports cleanly in the uWSGI context. Import failures crash the process on each poll cycle.

Segfault in C extension. If the spooler crashes with SIGSEGV (signal 11), a C extension called by the task handler is likely responsible. Check for memory corruption in C extension code invoked from the handler.

Tasks raising and requeuing

Poison tasks block the queue because the spooler retries them indefinitely. Options:

Fix the handler. The root cause is an exception in the task processing code. Identify it from application logs and fix the bug.

Move poison tasks aside. Manually move the offending files out of the spool directory to unblock the queue. This is a temporary fix while the handler bug is being addressed.

# CAUTION: Stop the spooler first, or accept the race with active spooler file operations.
# Moving files from under a running spooler can cause claim conflicts or missed tasks.
mkdir -p /var/spool/uwsgi-quarantine
find /var/spool/uwsgi/ -type f -mmin +60 -exec mv {} /var/spool/uwsgi-quarantine/ \;

Add a max-attempts mechanism. Track retry counts within the task payload or a sidecar store, and return SPOOL_OK (removing the file) after N retries to prevent infinite loops.

Spooler is slow

If task processing is inherently slow (heavy computation, slow external calls), increase parallelism or reduce per-task cost:

Multiple spooler processes. Use --spooler-processes <n> to run N spooler processes against the same spool directory. Each process independently scans and claims files, increasing throughput.

Reduce per-task cost. Profile the task handler and optimize the slowest operations. Add timeouts to external calls so a single slow dependency does not monopolize the spooler.

Adjust poll frequency. The default spooler poll frequency is 30 seconds, configurable via --spooler-frequency <secs>. If tasks are time-sensitive, reduce this value. More frequent polling increases filesystem load, which matters when thousands of files are present.

Use spooler-harakiri. --spooler-harakiri <secs> sets a timeout for spooler tasks. Without it, a single hung task blocks the spooler indefinitely, similar to how request workers hang without --harakiri.

Disk full

Clear old processed files (if your spooler leaves them behind) or move the spool directory to a larger filesystem. If the spooler moves failed tasks to a separate directory, that directory may also be filling up.

inotify limits

Increase the kernel inotify watch limit if the spooler relies on filesystem event notification:

# Current limit
cat /proc/sys/fs/inotify/max_user_watches

# Increase temporarily (requires root)
sysctl -w fs.inotify.max_user_watches=524288

# Persist across reboots
echo "fs.inotify.max_user_watches=524288" >> /etc/sysctl.conf

Other processes on the host share the inotify watch pool. If many services watch files, the spooler may lose out.

ext4 dir_index degradation

With thousands of files in a single spool directory, ext4 HTree directory indexing can degrade scan performance. The spooler must readdir the directory on every poll cycle, and large directories make this expensive.

Shard tasks across subdirectories. If using --spooler-ordered (priority-based scanning with numbered subdirectories), tasks are distributed across directories. Otherwise, consider sharding at the application level by writing tasks to date-prefixed or hash-prefixed subdirectories.

Clean up regularly. Ensure processed tasks are removed promptly.

Consider tmpfs. tmpfs eliminates disk I/O entirely but loses durability on restart. Use only if task loss on crash is acceptable.

Prevention

Monitor spoolers[].tasks as a trend, not a threshold. A static count of 50 pending tasks might be normal. Sustained growth is the warning sign. Track the delta over time and alert on the slope.
Monitor spoolers[].respawns. Any non-zero respawn rate for the spooler is abnormal. Unlike worker respawns (which include expected max-requests recycling), spooler respawns always indicate a crash.
Alert on oldest task age. Even if tasks is stable, a high oldest-task age means deferred work is delayed beyond its expected SLA.
Pre-create spool directories with correct ownership. Avoid the permission race where the master creates directories as root before dropping privileges.
Add max-attempts to task handlers. Poison tasks that retry forever block the queue. Implement a retry limit and return SPOOL_OK after exhaustion.
Watch disk usage on the spool filesystem. A filling disk will eventually prevent new task writes, causing silent task loss from the producer side.
Run multiple spooler processes if task volume warrants it. A single spooler process is a serialization bottleneck by design. --spooler-processes removes it.
Configure spooler-harakiri. Without a timeout, a single hung task blocks the spooler indefinitely. Set it to a reasonable multiple of your expected task duration.

How Netdata helps

Netdata collects the uWSGI stats server JSON and exposes spooler metrics alongside worker, memory, and request signals at per-second resolution. Useful correlations for spooler backlog:

spoolers[].tasks trend at per-second resolution. A backlog that grows during traffic spikes and drains during lulls indicates a capacity issue. Monotonic growth with no drain indicates a crashed or stuck spooler. Per-second collection captures patterns that 30-second polling misses.
spoolers[].respawns correlated with crash events. Spikes in spooler respawns, correlated with disk I/O metrics and system logs, help isolate the crash cause.
Disk I/O and filesystem fullness. Disk metrics show I/O wait, read/write rates, and filesystem usage on the partition holding the spool directory. Correlating disk saturation with rising tasks distinguishes “slow disk” from “slow handler.”
Worker pool health. If request workers are saturated, they may stop enqueuing tasks at the normal rate. This can mask a spooler backlog: tasks appears stable because production stopped, not because consumption improved.
Anomaly detection on task count. A slowly growing backlog can look normal in absolute terms but anomalous relative to historical patterns.