$ guides / postfix / postfix-queue-gridlock-slow-destination ▌

Operations Guides

Postfix queue gridlock: one slow destination stalls all mail

Mail delivery has stopped for everyone, but nothing looks broken. CPU is low. Network throughput is low. The active queue is full, the deferred queue is growing, and none of the usual suspects (DNS failure, content filter backpressure, disk exhaustion) explain it. This is Postfix queue gridlock: a single slow or throttling destination has consumed the active queue, and the queue manager’s fair scheduler is letting it starve every other destination.

The signature: active queue sits at or near qmgr_message_active_limit (default 20,000). The deferred queue grows steadily. Per-destination log analysis shows one or two domains dominating deferred entries with “connection timed out” or 4xx rate-limit responses. Delivery to healthy destinations has not failed, it has simply stopped being scheduled because no active queue slots are available.

What this means

Postfix’s queue manager (qmgr) is a fair scheduler, not a priority scheduler. It distributes delivery attempts across destinations within resource limits, and it does not deprioritize a destination just because that destination is slow. Every message to the congested destination occupies an active queue slot until it is delivered, deferred, or times out. When the destination is genuinely slow (network latency, greylisting, throttling), each slot is held longer, consuming a disproportionate share of the active queue.

flowchart TD
    A[Incoming mail to all destinations] --> B["Active queue
limit: 20,000 slots"]
    B --> C{qmgr fair scheduler}
    C -->|grabs slots| D["smtp to slow-domain
timing out"]
    C -->|starved, no slots| E["smtp to healthy-A
waiting"]
    C -->|starved, no slots| F["smtp to healthy-B
waiting"]
    D -->|4xx or timeout| G["Deferred queue
growing steadily"]
    B -->|no free slots| H["New mail stuck
in incoming queue"]

When enough messages to the slow destination accumulate, the active queue fills to its limit. The queue manager cannot schedule new deliveries to any destination. New mail stays in the incoming queue. The system appears idle from a resource perspective because the bottleneck is not CPU, network, or disk. It is active queue slot allocation.

The default qmgr_clog_warn_time is 300 seconds (5 minutes). When a destination clogs the active queue for longer than this threshold, qmgr emits a warning to the log. The warning names the destination, which is your fastest path to identifying the culprit.

Common causes

Cause	What it looks like	First thing to check
Destination rate limiting	Deferred entries with “4.7.1” or “rate limited” for one domain	Check log for 4xx codes from that domain’s MX
IP reputation throttling	Major provider (Gmail, Outlook) returning 4xx with throttling language	Check sender score and blocklist status
Network partition to destination	“connection timed out” to one domain, others healthy	Test connectivity to the destination MX
Destination infrastructure failure	One domain rejecting or timing out consistently	Check destination status pages, test from different network
Postfix concurrency too high for destination	Destination responds but throttles after burst of connections	Review `smtp_destination_concurrency_limit` vs destination tolerance

Quick checks

Run these read-only commands to confirm gridlock and identify the consuming destination.

# Active queue size vs limit
ls -1 /var/spool/postfix/active/ | wc -l
postconf -h qmgr_message_active_limit

# Deferred queue size
find /var/spool/postfix/deferred -type f | wc -l

# Which destination dominates deferred entries
grep 'status=deferred' /var/log/mail.log | awk -F'to=<|>,' '{print $2}' | cut -d@ -f2 | sort | uniq -c | sort -rn | head

# Current concurrency settings
postconf -h smtp_destination_concurrency_limit initial_destination_concurrency

# Queue shape by domain (incoming + active by default)
qshape | head -20

# Queue shape for active queue only, with sender stats
qshape -s active | head -20

# Look for clogging warnings from qmgr
grep -i 'clog' /var/log/mail.log | tail -10

# Spot-check sent vs deferred counts for a recent minute
grep "$(date -d '5 minutes ago' '+%b %e %H:%M')" /var/log/mail.log | grep -c 'status=sent'
grep "$(date -d '5 minutes ago' '+%b %e %H:%M')" /var/log/mail.log | grep -c 'status=deferred'

The log file path differs by distribution. Debian and Ubuntu use /var/log/mail.log; RHEL, CentOS, and Fedora use /var/log/maillog.

How to diagnose it

Confirm active queue saturation. Compare ls -1 /var/spool/postfix/active/ | wc -l against postconf -h qmgr_message_active_limit. If the ratio is above 80 percent, you are in or approaching gridlock.
Identify the consuming destination. The deferred-log grep above gives you the domain. Cross-check with qshape, which breaks down the active and incoming queues by recipient domain and message age. The domain with the most messages and the oldest average age is your culprit.
Verify the destination is actually slow, not just slow for you. Test connectivity directly. If the destination returns 4xx codes with throttling language, the issue is rate limiting or reputation. If connections time out entirely, you have a network path problem. If the destination is healthy from another network but not from yours, suspect local firewall, NAT, or IP-level blocking.
Confirm healthy destinations are starving, not failing. Check deferred entries for domains other than the culprit. If healthy domains have zero or near-zero deferred entries but also zero recent deliveries, they are starved by slot exhaustion, not failing on their own.
Check qmgr clogging warnings. If qmgr_clog_warn_time is at its default of 300 seconds and the condition has persisted, qmgr should have logged a clogging warning naming the destination.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Active queue size vs limit	Tells you whether the scheduler can admit new deliveries	Above 80 percent of `qmgr_message_active_limit` for more than 10 minutes
Deferred queue growth rate	Velocity, not absolute size, indicates acute failure	Sustained growth exceeding 1,000 messages/hour
Per-destination deferred count	Identifies which destination is consuming slots	One domain accounts for more than 50 percent of deferred entries
Delivery rate vs injection rate	Confirms throughput collapse vs backlog	Delivery rate below 50 percent of injection rate for more than 5 minutes
Per-destination SMTP connection latency	Distinguishes slow destination from network failure	Connection setup exceeding 10 seconds consistently for one domain
qmgr clogging warnings	Direct signal that the scheduler has identified the problem	Any occurrence

Fixes

Reduce destination concurrency globally

The fastest emergency lever is lowering smtp_destination_concurrency_limit:

# Reduce concurrency for all smtp destinations
postconf -e 'smtp_destination_concurrency_limit=5'
postfix reload

This helps if the destination is throttling because Postfix is opening too many parallel connections. But it affects all destinations, not just the problem one. Healthy destinations that were delivering fine at the previous concurrency will also slow down. Use this as a stopgap while you set up a transport-map isolation.

Isolate the slow destination with a transport map

The proper fix is a dedicated transport for the problematic domain with reduced concurrency. This preserves full concurrency for healthy destinations.

Add to master.cf:

slow unix - - n - - smtp -o syslog_name=postfix-slow

Add to main.cf:

slow_destination_concurrency_limit = 2
slow_destination_recipient_limit = 5

Add to your transport map (typically /etc/postfix/transport):

problem.domain    slow:

Then apply:

postmap /etc/postfix/transport
postfix reload

New mail to the problem domain routes through the slow transport with limited concurrency. Existing queued messages will need to be re-queued to pick up the new transport mapping: postsuper -r ALL. This is disruptive on large queues. Consider re-queueing in batches by queue ID.

A common misconfiguration trap: setting destination_recipient_limit to 1 changes the meaning of destination_concurrency_limit from per-domain to per-recipient. If you need per-recipient concurrency control, set it explicitly and understand the semantic shift. Otherwise leave it at its default.

Hold and release

For immediate relief when the active queue is completely jammed, move the problem domain’s messages to the hold queue. This frees active queue slots for healthy destinations instantly.

# Hold messages addressed to the problem domain across all queues
postqueue -p | awk 'BEGIN{RS=""} /@problem\.domain/{gsub(/[*!]$/,"",$1); print $1}' | postsuper -h -

This parses postqueue -p output, matching the recipient domain in each message block. The gsub strips the * (active) or ! (held) marker from the queue ID before passing it to postsuper. Verify the hold count matches expectations before proceeding.

After the destination recovers, release the held mail:

# Release all held messages (may cause a delivery burst)
postsuper -H ALL

Releasing all held mail at once can overwhelm the destination. If you held a large batch, release in smaller groups or verify the destination can handle the volume first.

Do not increase concurrency as a reflex

The Postfix tuning documentation is explicit: reflexive increases to concurrency parameters in the face of congestion can make problems worse. If a destination is throttling your connections, increasing concurrency sends more connections that the destination will reject or rate-limit, worsening the feedback loop. Reduce concurrency or isolate the destination instead.

Prevention

Monitor active queue utilization, not just total queue depth. Track active_queue_size / qmgr_message_active_limit as a ratio. Alert above 80 percent.
Track per-destination deferred counts. A single domain climbing past 50 percent of total deferred entries is an early warning.
Pre-configure transport maps for known rate-limited destinations. Major providers that enforce strict per-IP connection limits should have dedicated transports before they become a problem.
Understand your destination tolerance. If you send high volume to Gmail, Outlook, or a corporate partner, know their published rate limits and configure concurrency accordingly.
Keep qmgr_clog_warn_time at or below its default. The warning is your fastest automated signal that a destination is monopolizing the active queue.

How Netdata helps

Per-second active queue depth lets you see slot exhaustion as it develops, not minutes later when delivery has already stalled.
Deferred queue growth rate computed from per-second samples gives you velocity, not just snapshots, so you can distinguish a transient blip from sustained accumulation.
Delivery rate vs injection rate correlation makes throughput collapse immediately visible. When the two diverge, something is blocking delivery, and the timing lines up with queue depth changes.

Per-destination deferred visibility surfaces which domain dominates deferred entries, pointing you at the culprit before you need to run manual greps.

qmgr clogging warning detection in log streams provides an automated alert the moment the scheduler identifies a monopolizing destination.
Correlation across signals (active queue saturation, deferred growth, delivery rate drop, single-domain deferral spike) turns a confusing “mail is slow but nothing looks broken” symptom into a clear gridlock diagnosis.