HAProxy run queue rising: task backlog and scheduler pressure

HAProxy’s event loop processes every connection, request, health check, and timer as tasks scheduled across its threads. When tasks arrive faster than threads can drain them, they pile up in the run queue. That backlog is the earliest internal signal that HAProxy is CPU-saturated: it appears before latency spikes, before backend queuing, and before client timeouts.

It is also the signal to use when Idle_pct is meaningless. With busy-polling enabled, the event loop never sleeps, so Idle_pct sits near zero by design. The Run_queue field from show info and the per-thread counters from show activity are the replacement.

A rising run queue does not tell you why the scheduler is behind. The usual drivers are TLS handshake load, expensive ACL or regex evaluation, Lua scripts, and compression. The job is to confirm the backlog, find which threads and which work items cause it, and relieve the specific CPU consumer.

What this means

HAProxy multiplexes hundreds of thousands of connections across a small number of threads (nbthread). Each thread runs a poll loop: wait for I/O and timer events, wake the tasks that became runnable, execute them, repeat. Tasks are connection state machines, health checks, peers, stick-table expiry, and internal housekeeping.

Two fields in show info describe the scheduler’s position:

• Tasks: total active tasks. Scales with connections and configured health checks. A size gauge, not a pressure gauge.
• Run_queue: tasks that are runnable but waiting for a thread. Healthy operation is zero or near zero. A sustained nonzero, growing value means work is arriving faster than threads can execute it.

show activity adds roughly 32 per-thread counters: loop iterations, poll timing, run-queue depth per thread. Two things matter: aggregate backlog, and skew. One thread carrying most of the work caps overall throughput even when the average looks fine.

The failure cascade:

flowchart TD
  A[CPU driver: TLS handshakes, ACL/regex, Lua, compression] --> B[Tasks stay runnable longer per loop]
  B --> C[Run_queue rising, per-thread backlog]
  C --> D[Slower accept and request dispatch]
  D --> E[Client-visible latency climbs]
  D --> F[Backend ctime rises, accept slows]
  E --> G[Timeouts, client aborts]
  C --> H[Kernel accept queue overflows - ListenOverflows]
  H --> I[SYNs dropped before HAProxy sees them]

The kernel accept queue step is the nasty part. When the event loop is too busy to call accept() promptly, new SYNs overflow net.core.somaxconn and are dropped silently. HAProxy’s own metrics show nothing wrong because those connections never reached user space.

Common causes

Quick checks

All read-only, via the runtime socket. Adjust the socket path for your deployment.

# 1. Scheduler position: run queue, tasks, idle
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | \
  grep -E "^(Tasks|Run_queue|Idle_pct|Uptime_sec):"

# 2. Per-thread activity counters (HAProxy 1.9+)
echo "show activity" | socat unix-connect:/var/run/haproxy.sock stdio

# 3. TLS handshake load (the usual prime suspect)
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | \
  grep -E "^(SslRate|SslFrontendKeyRate|SslCacheLookups|SslCacheMisses|CurrSslConns):"

# 4. Traffic volume for correlation
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | \
  grep -E "^(ConnRate|SessRate|CurrConns):"

# 5. Kernel accept queue overflow (event loop too busy to accept)
nstat -az | grep -i listen

# 6. System view: is HAProxy the CPU consumer, and on which cores
# (master-worker setups have multiple PIDs; include all of them)
top -H -p "$(pgrep -x haproxy | paste -sd,)"

# 7. Is compression active and busy
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "FRONTEND" {print $1": comp_in="$52" comp_out="$53" comp_rsp="$55}'

# 8. Two samples of Run_queue 10 seconds apart to see the trend
for i in 1 2; do
  echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep "^Run_queue:"
  sleep 10
done

Notes on interpretation:

• A single Run_queue snapshot means little. Sample repeatedly: a backlog that recovers between samples is burst absorption; a backlog that grows monotonically is saturation.
• show activity counters are 32-bit and wrap on busy instances. Compare deltas over short windows, not absolute values from long-running processes.
• show activity exists since HAProxy 1.9. On 2.7 and later, output shows an aggregated first column with per-thread values in brackets; older versions show per-thread values only.

How to diagnose it

1. Confirm the backlog is real. Take three Run_queue samples 5-10 seconds apart. Sustained nonzero values, especially rising, confirm scheduler pressure. Record Tasks and Idle_pct (if not busy-polling) for the same samples.

2. Check whether you are in busy-polling mode. If busy-polling is set in the global section, disregard Idle_pct for this entire diagnosis. It reads near zero on a perfectly healthy process. Run queue and show activity are your only in-process saturation signals; per-core system CPU is the external check.

3. Look for thread skew in show activity. Compare per-thread run-queue depth and loop counters. One thread far ahead of the others means an affinity or distribution problem (for example, one listener thread absorbing all accepts). Uniform backlog across all threads means a global CPU deficit instead.

4. Watch the loop timing counters. In show activity, poll loop duration tells you whether each iteration is getting slower (tasks doing more work per pass) or the queue between passes is getting deeper (more tasks waking per pass). Slower iterations point at expensive per-task work: regex ACLs, Lua, compression. Deeper queues at normal iteration speed point at volume: handshake storms, connection floods.

5. Identify the CPU driver. Walk the usual suspects in order of likelihood:

   • TLS: is SslFrontendKeyRate elevated versus baseline? Is the session cache effective (SslCacheMisses / SslCacheLookups)? A broken or undersized cache forces a full handshake on every connection.
   • Request-phase CPU: does the backlog track req_rate (per-request work: ACLs, regex, Lua) or ConnRate/SslFrontendKeyRate (per-connection work: TLS)?
   • Compression: if enabled, is a large share of responses being compressed while the proxy is CPU-bound?

6. Rule out the invisible loss path. Check ListenOverflows/ListenDrops and the ConnRate vs SessRate gap. If the event loop is too busy to accept, clients are already timing out at SYN and you will not see them in any HAProxy counter.

7. Correlate with downstream symptoms. Backend ctime rising with low econ, and frontend latency rising while backend rtime stays flat, is the signature of HAProxy-side CPU saturation rather than a backend problem. If rtime is rising too, you may be looking at a backend slowdown instead: see the 504 cascade guide linked below.

Metrics and signals to monitor

Fixes

TLS handshake load

• Fix session resumption first. If SslCacheMisses is high relative to lookups, the cache is too small or tickets are being invalidated (for example by frequent reloads). Restoring resumption removes most of the handshake CPU cost immediately and is the cheapest real fix.
• Reduce new-connection churn. Long keep-alive lifetimes and HTTP/2 amortize handshakes across many requests. Check whether clients or an upstream device are forcing one connection per request.
• Add threads or offload. If the load is genuine growth, raise nbthread toward available cores (requires a reload; plan it) or move termination to a dedicated layer. Adding threads during an active saturation event is disruptive: treat it as a planned change, not a hot fix.

Expensive per-request work (ACLs, regex, Lua, compression)

• Simplify hot-path ACLs. Regex matching on every request multiplies cost by request rate. Prefer exact/string or map-based lookups where possible.
• Audit Lua actions. Blocking or long-running Lua in http-request/http-response rules stalls the calling thread. Move work out of the request path or make the actions cheaper.
• Restrict compression scope. Compress only compressible content types, or disable compression while CPU-bound. Compression trades CPU for bandwidth; when CPU is the scarce resource, the trade is wrong.

Thread imbalance

• If show activity shows one thread doing most of the work, review listener binding and thread affinity configuration, and consider SO_REUSEPORT-based distribution. Verify with per-thread counters after the change; averages will hide the fix if it did not work.

When you cannot reduce the work

• Shed or shape load. Rate limiting at the frontend (rate-limit sessions, stick-table based limits) caps the arrival rate so the backlog stops growing. This protects existing sessions at the cost of rejecting new ones.
• Do not restart HAProxy as a first response. A restart drops all connections and resets all counters, destroying the evidence and briefly worsening client impact. If the process is healthy but saturated, the fix is less work or more capacity, not a new process.

Prevention

• Baseline the run queue. Record typical Run_queue behavior at daily peak so “rising” has meaning for your workload. The norm is zero or near zero; know your bursts.
• Alert on the busy-polling-appropriate signal. If busy-polling is on, disable Idle_pct alerting and alert on sustained Run_queue > 0 or per-core CPU of HAProxy threads instead. If it is off, alert on Idle_pct < 20% sustained as the early warning and Run_queue as confirmation.
• Watch the CPU drivers, not just the symptom. Trend SslFrontendKeyRate, cache miss ratio, and compressed-response volume. These move before the run queue does.
• Handle counter wrap. show activity counters are 32-bit. On high-traffic instances they wrap within the process lifetime; collect frequently, compute deltas, and treat resets on reload as new baselines (Uptime_sec tells you when a reload happened).
• Capacity-plan from peak loop headroom. When peak-hour Idle_pct trends below ~30% (non-busy-polling) or the run queue starts appearing at peak regularly, you are inside the headroom buffer. Scale before the backlog becomes continuous.
• Keep net.core.somaxconn and the SYN backlog sized for burst so a transiently busy loop absorbs connection bursts instead of silently dropping SYNs.

How Netdata helps

• Netdata collects HAProxy show info and stats CSV fields continuously, so Run_queue, Tasks, Idle_pct, and SslFrontendKeyRate are graphed at high resolution instead of sampled by hand during the incident.
• Per-second collection makes the rate of rise visible, which is what separates burst absorption from true saturation; a single manual sample cannot show that.
• Correlating SslFrontendKeyRate and SSL cache counters against the run queue on one dashboard answers the “is this TLS?” question in seconds rather than a socket session.
• Backend ctime, rtime, and econ alongside the scheduler signals confirm the backlog is HAProxy-side CPU pressure and not a backend slowdown masquerading as one.
• System-level metrics (per-core CPU, ListenOverflows) collected from the same host close the loop on thread skew and the silent SYN-drop path.
• Anomaly detection on the run queue and handshake rate catches slow-building scheduler pressure during off-hours, before it coincides with peak traffic.

Related guides

• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy connect time (ctime) high: network latency and backend accept-queue overflow
• HAProxy connection errors (econ): backend connections refused, timed out, or reset
• HAProxy 502 Bad Gateway: backend connection and response failures
• HAProxy 503 Service Unavailable: no server is available to handle this request
• HAProxy 5xx delta: telling HAProxy-generated errors from backend errors
• HAProxy backend losing servers: active server count and cascade risk
• HAProxy health checks green but the application is broken: when UP does not mean healthy
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable
• HAProxy health check L7STS: server DOWN on the wrong HTTP status