$ guides / postfix / postfix-inode-exhaustion ▌

Operations Guides

Postfix inode exhaustion: the queue filesystem blind spot behind No space left on device

Postfix reports No space left on device in your logs. You check df -h /var/spool/postfix and see 50% free space. Mail acceptance is failing and queue operations are erroring. The obvious explanation does not match reality.

The cause is almost always inode exhaustion on the queue filesystem. Postfix creates one file per queue entry: every incoming message, every deferred retry, every bounce notification. Each file consumes one inode regardless of how few bytes it occupies. A deferred queue with 500,000 tiny messages might use only a couple of GB of block space but exhaust every inode on the partition.

The fix is to clean the queue and address the underlying growth cause. The prevention is to monitor df -i explicitly, because monitoring df -h alone will never warn you about this failure mode.

What this means

Postfix stores mail in multiple queues under /var/spool/postfix/: maildrop/, incoming/, active/, deferred/, hold/, corrupt/, bounce/, and defer/. Every message in every queue is a separate file on disk. Postfix also writes per-message metadata files in defer/ and bounce/ for delivery status records. A single message that fails delivery and generates a bounce can create files across three or four queue subdirectories.

When the filesystem runs out of inodes, the kernel returns ENOSPC on file creation. Postfix logs this as No space left on device (specifically, mail_queue_enter: create file maildrop/xx.xx: No space left on device or similar). The error string is identical whether the filesystem ran out of blocks or inodes, which is why checking df -h alone is misleading.

On ext4, the inode pool is fixed at filesystem creation time. The default bytes-per-inode ratio allocates roughly one inode per 16 KiB of filesystem space. A 20 GB partition gets approximately 1.3 million inodes. A Postfix server accumulating bounce messages or a runaway deferred queue can exhaust that faster than you would expect.

On XFS, inodes are allocated dynamically up to a configurable percentage of the filesystem. This makes XFS more resistant to inode exhaustion than ext4, but it is not immune. A sufficiently large queue will still hit the ceiling.

The degradation is cliff-edge, not gradual. Operations succeed until the last inode is consumed, then every file creation fails immediately.

flowchart TD
    A[Mail accepted faster than delivered] --> B[Deferred queue grows]
    B --> C[Bounce and defer metadata files accumulate]
    C --> D[Inode count climbs toward ceiling]
    D --> E{Inodes exhausted?}
    E -- No --> B
    E -- Yes --> F[creat and open return ENOSPC]
    F --> G[Postfix logs: No space left on device]
    G --> H[Queue file creation fails]
    G --> I[PID and lock file writes fail]
    G --> J[df -h still shows free space]

Common causes

Cause	What it looks like	First thing to check
Runaway deferred queue	Delivery to one or more destinations failing; deferred queue growing rapidly	`find /var/spool/postfix/deferred -type f \| wc -l` and deferral reasons in mail.log
Bounce storm	Massive bounce generation to forged or invalid senders; MAILER-DAEMON dominating the queue	`grep 'status=bounced' /var/log/mail.log \| tail -20` and queue sender distribution
Double-bounce loop	Bounces of bounces multiplying; metadata files in bounce/ and defer/ growing fast	`postqueue -p \| grep MAILER-DAEMON \| wc -l` and `find /var/spool/postfix/bounce -type f \| wc -l`
Small ext4 partition with default inode density	Partition sized correctly for bytes but inode count too low for Postfix workload	`df -i /var/spool/postfix` showing high IUse% relative to expected capacity
Stale queue never cleaned	Old deferred messages accumulating over weeks or months, never expiring	`find /var/spool/postfix/deferred -type f -mtime +5 \| wc -l` compared to `maximal_queue_lifetime`

Quick checks

These commands are read-only and safe to run at any time.

# Check inode usage on the queue filesystem
df -i /var/spool/postfix

# Compare with block space usage (often diverges dramatically)
df -h /var/spool/postfix

# Global inode state (first number: allocated, second: free)
cat /proc/sys/fs/inode-nr

# Total file count across all queue subdirectories
find /var/spool/postfix -type f 2>/dev/null | wc -l

# Per-subdirectory counts to identify which queue is hoarding inodes
for d in maildrop incoming active deferred hold corrupt bounce defer; do
  count=$(find "/var/spool/postfix/$d" -type f 2>/dev/null | wc -l)
  printf "%-12s %s\n" "$d" "$count"
done

# Look for the specific Postfix error in recent logs
grep -i 'No space left on device\|not enough free space' /var/log/mail.log | tail -20

# Check deferred queue growth rate (compare two snapshots 60 seconds apart)
find /var/spool/postfix/deferred -type f | wc -l && sleep 60 && find /var/spool/postfix/deferred -type f | wc -l

# Verify queue manager is still responsive (may be slow on huge queues)
time postqueue -p | tail -5

How to diagnose it

Confirm inode exhaustion is the cause. Run df -i /var/spool/postfix. If IUse% is at or near 100%, inodes are the problem. If df -h simultaneously shows adequate free space, you have confirmed the divergence.
Identify which queue is consuming inodes. Run the per-subdirectory count loop from the quick checks. The deferred/ directory is the most common culprit, but bounce/ and defer/ can also accumulate large numbers of small metadata files. Postfix hashes entries into subdirectories (deferred/0/ through deferred/F/), so check nested directories too.
Check for subdirectory-level performance cliffs. A single queue subdirectory with more than 10,000 entries can degrade directory operations on ext4, making postsuper cleanup slow even before inodes are fully exhausted. Run for d in /var/spool/postfix/deferred/*/; do echo "$(find "$d" -type f | wc -l) $d"; done | sort -rn | head to spot hot hash buckets.
Determine the root cause of queue growth. Check mail.log for deferral reasons: grep 'status=deferred' /var/log/mail.log | tail -50. Look for patterns such as a single slow destination monopolizing delivery, DNS failures causing universal deferrals, or bounce storms generating massive local submissions. The qshape utility (shipped with Postfix) breaks down queue depth by recipient domain and is useful here: qshape deferred | head -20.
Estimate runway. If inodes are not yet exhausted but trending upward, calculate how many hours remain at the current growth rate. Subtract free inodes from the growth rate observed in step 2 of the quick checks. If the rate is accelerating due to retry backoff compounding, treat the estimate as optimistic.
Check for double-bounce loops. If bounce/ and defer/ are the dominant consumers, look for MAILER-DAEMON messages multiplying in the queue. Double-bounces are discarded by default, but a misconfigured bounce pipeline can create metadata files faster than they are cleaned.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Inode usage on queue filesystem (`df -i`)	Direct indicator of the failure mode	>80% used, or fewer than 10,000 free inodes
Block space usage (`df -h`)	Often diverges from inode usage; monitoring this alone misses the problem entirely	Normal while inode usage is critical
Total queue file count	Proxy for inode consumption; correlates with growth rate	>50% of filesystem inode capacity
Deferred queue size	Primary accumulator of inodes in most incidents	Sustained growth exceeding 100 files/hour
Per-subdirectory entry counts	Detects ext4 directory performance cliff before full exhaustion	Any single subdirectory exceeding 10,000 entries
Deferred queue growth rate (velocity)	Leading indicator; rate of change predicts time to exhaustion	Sustained positive growth over 4 or more hours
`No space left on device` errors in mail.log	Confirms Postfix has hit the wall	Any occurrence means active file creation failure
Bounce rate	Bounce storms create secondary inode consumers	>1% sustained, or sudden 10x spike from baseline

Fixes

Emergency: drain the consuming queue

If mail acceptance has stopped, the immediate priority is freeing inodes. The deferred queue is almost always the largest consumer.

# Count what you are about to delete
find /var/spool/postfix/deferred -type f | wc -l

# WARNING: deletes ALL deferred mail. Irreversible. Use only in emergency.
postsuper -d ALL deferred

Deleting deferred messages with postsuper cascades to their associated defer/ and bounce/ metadata records. If orphaned metadata (message files already gone, stale defer/bounce entries left behind) is the problem, run postsuper -s to clean up structural inconsistencies.

postsuper -d ALL without a queue name deletes all messages across all queues (maildrop, incoming, active, deferred, hold). This is destructive and irreversible.

If you need to preserve mail but stop retry-driven growth, hold the queue instead of deleting:

# Hold all deferred mail (preserves messages, stops retry scheduling)
postsuper -h ALL deferred

Holding does not free inodes. It only stops the qmgr from generating additional retry-related metadata. You still need to delete or deliver the held mail to reclaim inodes.

Cleanup is slow under pressure

postsuper -d is I/O-bound and single-threaded. On a filesystem near inode exhaustion, deletion itself is slow because the kernel must update directory structures and inode bitmaps for every removed file. A cleanup of hundreds of thousands of files can take hours. This is why maintaining >20% free inodes (50% preferred) matters: the headroom is needed for cleanup operations to run at reasonable speed. Do not wait until you are at 99% to start cleaning.

Fix the root cause of queue growth

Deleting the queue without addressing why it grew is a temporary fix. Common root causes:

Slow destination monopolizing delivery: reduce smtp_destination_concurrency_limit or use transport maps to isolate the problem destination. See Postfix deferred queue growing: why mail piles up and how to drain it.
DNS resolver failure causing universal deferrals: fix the resolver, then let the queue drain naturally as retries succeed. See Postfix DNS resolver failure: when a broken resolver defers mail to everyone.
Bounce storm from backscatter: tighten recipient restrictions to reject unknown recipients before queueing. See Postfix backscatter storm: bounces to forged senders and blocklisting.
Double-bounce loop: check for MAILER-DAEMON mail multiplying in the queue. See Postfix double-bounce loop: MAILER-DAEMON mail multiplying in the queue.
Content filter backpressure: if Amavis or Rspamd is slow, mail piles up in the incoming queue before it ever reaches active or deferred. See Postfix content_filter backpressure: incoming queue growth when Amavis or Rspamd slows.

Long-term: increase inode capacity

If the filesystem is structurally too small for the Postfix workload, cleanup alone will not prevent recurrence.

On ext4: the inode count is fixed at mkfs time. You cannot increase it without reformatting. When provisioning a new queue partition, use mkfs.ext4 -T small or specify bytes-per-inode explicitly (-i) to increase inode density.
On XFS: inodes are allocated dynamically up to a configurable percentage. Verify the allocation ceiling with xfs_info and increase it if needed. XFS is more forgiving than ext4 but still monitor.
Separate the queue filesystem: place /var/spool/postfix on its own partition or logical volume so that other filesystem consumers (logs, package caches) do not compete for the same inode pool.

Prevention

Monitor df -i on the queue filesystem explicitly. This is the single most important change. Standard disk space monitoring will never catch this failure mode. Alert at 80% inode usage, page at 90%.
Track deferred queue growth rate, not just absolute size. A deferred queue of 10,000 messages that is shrinking is healthy. The same 10,000 growing at 500 messages/hour is a crisis in progress. Alert on sustained positive growth over 4-hour windows.
Set per-subdirectory monitoring. Alert when any single queue subdirectory exceeds 10,000 entries. This catches ext4 directory performance degradation before it turns into full exhaustion.
Automate periodic queue cleanup. For high-volume servers, schedule regular removal of deferred mail past maximal_queue_lifetime (default 5 days). This prevents slow accumulation from eventually hitting the wall.
Size the queue filesystem for inodes at provisioning time. Calculate expected peak queue depth during worst-case destination failure, multiply by 4 (for metadata files across subdirectories), and ensure the filesystem has enough inodes for that plus 50% headroom.
Keep >20% free inodes, 50% preferred. The headroom is not just for safety margin. postsuper -d cleanup needs working space to operate efficiently. Near exhaustion, cleanup itself becomes painfully slow.

How Netdata helps

Netdata collects the signals that distinguish inode exhaustion from a genuine disk-full condition:

Filesystem inode metrics are collected per-second per mount point, showing inode consumption trending toward the ceiling before the cliff.
Block space and inode metrics appear together in the same dashboard, making the divergence immediately visible: block usage normal while inode usage spikes.
Postfix queue size metrics provide a proxy for total inode consumption. Combined with Netdata’s rate calculation, this enables velocity-based alerting on queue growth.
ML-based anomaly detection on inode usage rate can flag slow, chronic accumulation that precedes acute exhaustion, even when absolute counts remain below static thresholds.

For inode-specific alerting, configure an alarm on the filesystem inode utilization metric for the queue filesystem mount point.