HAProxy hot-thread skew: when average Idle_pct hides a saturated thread

Latency is climbing, throughput has plateaued below what the hardware should deliver, and show info reports Idle_pct at a comfortable 60 or 70 percent. System CPU shows one core pegged while the others coast. The bottleneck is not HAProxy as a whole. It is one thread.

With nbthread > 1 (the default since HAProxy 2.0, where nbthread is set to the number of available CPUs), work is spread across per-thread event loops. Idle_pct is a process-level average across all of them. One thread pinned at 100% while the rest sit at 80% idle averages out to “looks fine,” but that hot thread caps overall throughput: every connection it owns gets processed late, and no other thread can take them over.

This guide covers how to confirm the skew with show activity, what usually causes it, and how to fix the distribution.

What this means

Each HAProxy thread runs its own scheduler and event loop. Connections are assigned to threads at accept time, and under normal conditions the load across threads should be roughly even. Thread skew means one thread is doing a disproportionate share of the work: more loop iterations, more accepts, more tasks, longer run-queue waits.

The practical consequences:

• Throughput ceiling. A single hot thread caps overall throughput no matter how much headroom the other threads have.
• Latency concentration. Connections owned by the hot thread see queuing and scheduling delay while connections on other threads are fine. Aggregate latency averages hide this the same way Idle_pct does.
• Misleading saturation signals. Alerting on Idle_pct never fires, because the average stays healthy.

The two reliable windows into per-thread behavior are the show activity runtime command (per-thread event counters, available since HAProxy 1.9) and show threads (per-thread scheduler state). Neither appears in the CSV stats, so most monitoring stacks never collect them.

flowchart TD
  A[Idle_pct healthy, latency rising] --> B{nbthread > 1?}
  B -- no --> C[Classic CPU saturation path]
  B -- yes --> D[show activity per-thread deltas]
  D --> E{One thread dominates loops / accepts / run queue?}
  E -- no --> F[Even load, look elsewhere]
  E -- yes --> G[Hot thread found]
  G --> H[Check accept distribution, Lua, version bugs]

Common causes

Quick checks

All of these are read-only runtime API commands.

# Confirm thread count and the misleading average
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep -E "^(Idle_pct|Nbthread|Run_queue|Tasks):"

# Dump per-thread activity counters (default: all threads)
echo "show activity" | socat unix-connect:/var/run/haproxy.sock stdio

# On HAProxy 2.7r1 and later: aggregated view only
echo "show activity 0" | socat unix-connect:/var/run/haproxy.sock stdio

# On HAProxy 2.7r1 and later: a single thread
echo "show activity 3" | socat unix-connect:/var/run/haproxy.sock stdio

# Per-thread scheduler state; look for the '>' no-progress marker
echo "show threads" | socat unix-connect:/var/run/haproxy.sock stdio

On the OS side, per-core CPU confirms what the counters imply:

# Per-core utilization: one core pegged, others idle = skew signature
mpstat -P ALL 1 3

Three things to know before reading show activity output:

• Counters are 32-bit and wrap. On a busy instance, counters like loop iterations and context switches can wrap within hours. Single absolute values are meaningless; always take two samples and compute deltas.
• The counters are intentionally undocumented. The management guide states the fields are purposely not documented so their exact meaning is verified in the source where they are fed. Treat per-counter semantics as approximate and cross-check against the source or community knowledge for your version.
• Since HAProxy 2.7r1, show activity accepts an argument: a thread number for one thread, 0 for the aggregated column, or -1 for all threads (the default). The first output column is the total or average; per-thread values follow in square brackets in thread order.

How to diagnose it

1. Confirm the symptom is real. Idle_pct healthy, mpstat -P ALL shows one core saturated, and latency is degrading. If all cores are busy, this is classic CPU saturation, not skew.

2. Baseline the activity counters. Run show activity, wait 30 to 60 seconds under representative load, run it again. Compute per-thread deltas. The delta math still works on a wrapped counter if you take the difference modulo 2^32, but it is simpler to reset and sample from zero. If you do reset, do it deliberately and once; resets are process-wide, so do not put them in a collection loop.

3. Identify the dominant counters. Look for the thread whose deltas dwarf the others: loop iterations, accept events, poll-wait time, run-queue depth. On versions that expose it, per-thread CPU percentages in the output (for example an avg_cpu_pct row with one bracketed value far above the rest) confirm the skew directly.

4. Check for a stuck thread. Run show threads twice a few seconds apart. A thread marked with > made no progress between invocations; between two threads that pattern indicates a deadlock, on a single thread a corrupted list. That is not skew, it is a hang, and the process needs restarting.

5. Classify the skew type. If the hot thread’s advantage is mostly accept-side events, suspect connection distribution (step 6). If it is task or loop activity without matching accepts, suspect something executing on that thread: Lua, a stuck task, or a scheduler bug.

6. Test accept distribution. SO_REUSEPORT lets the kernel distribute incoming connections across threads sharing a listen socket, but the distribution is not guaranteed to be fair for every workload. The HAProxy management guide calls this out: noreuseport (or the -dR command-line flag) may be applied in multi-threading scenarios where load distribution issues are observed among the haproxy threads. Re-test with it disabled and compare deltas again. This changes listener behavior, so roll it out like any config change and expect a brief redistribution when it takes effect.

7. Rule out version bugs. Check your exact version against the changelog. Extreme skew with orders-of-magnitude asymmetry (one thread showing millions of run-queue events while Idle_pct still reads healthy) and no config explanation has been a scheduler-bug pattern; there is at least one report of a task repeatedly waking itself, fixed in 2.7.5. If you are on an affected version, upgrading is the first fix to try.

Metrics and signals to monitor

Two caveats that change interpretation:

• busy-polling. With busy-polling enabled, Idle_pct sits near zero by design because the loop never sleeps. In that mode, use run-queue depth from show activity or system CPU, never Idle_pct.
• Reloads. Activity counters are process-local and reset on reload. After a reload your baselines are gone.

Fixes

Even out accept distribution

If the skew is accept-driven, test noreuseport in the global section (or -dR at startup) so threads share a single listen socket instead of per-thread SO_REUSEPORT sockets. Measure before and after with show activity deltas; the right answer is workload-dependent. thread-groups can partition threads for NUMA alignment, and recent releases have improved automatic CPU binding. If threads are fighting over shared cores or crossing NUMA boundaries, explicit affinity is worth checking.

Remove serialized work

Anything that serializes threads will manufacture a hot thread. Lua execution uses a global lock, so Lua-heavy configurations serialize all threads behind whoever holds it; the fix is reducing or restructuring Lua work, not adding threads. The same logic applies to any expensive per-request work concentrated on one path.

Upgrade past known scheduler bugs

Extreme, config-independent skew (one thread with millions of events, others near zero) has been a bug pattern at least once, fixed in 2.7.5. If you are on an affected version, upgrade. nbproc is gone since 2.5, so multi-process workarounds from older guides are not available on current versions.

Restart a genuinely stuck thread’s process

If show threads shows a no-progress thread, that is a deadlock or corrupted list, not a distribution problem. Restart the process. Do not treat this as routine; it is a last resort after confirming the > marker persists across invocations.

Prevention

• Collect show activity deltas in your monitoring, not just show info. It is the only per-thread signal HAProxy exposes, and it sits in the “expert” tier of most monitoring maturity models because nobody collects it until they need it.
• Alert on skew, not on the average. A ratio like “max per-thread loop delta / median per-thread loop delta” sustained above several times is a usable alert. Absolute thresholds are not.
• Baseline after every config or version change that touches threading: nbthread, thread-groups, CPU affinity, noreuseport, Lua.
• Handle counter wrap in collection. Sample on a fixed interval short enough that 32-bit wrap cannot produce ambiguous deltas on your busiest instance, and treat counter resets as an operator action, not an automated one.
• Keep per-core CPU dashboards for HAProxy hosts. The OS sees the skew even when HAProxy’s own average hides it, and correlating the two is the fastest confirmation path.

How Netdata helps

• Netdata collects per-core CPU on HAProxy hosts, so the “one core pinned, average looks fine” signature is visible without any HAProxy-specific configuration.
• HAProxy stats collection surfaces event-loop and saturation signals alongside frontend and backend metrics, so you can correlate a latency rise on a specific frontend with proxy-level backlog in one view.
• Because Netdata keeps per-second resolution, the short sampling windows that thread-skew diagnosis needs (two samples, compute deltas) are already there; you do not have to reconstruct them from 1-minute scrapes.
• ML anomaly detection on per-core CPU and request-rate metrics flags the asymmetric pattern (one core anomalous, total traffic flat) that thread skew produces, which is the kind of divergence static thresholds on averages miss.
• Correlating HAProxy latency signals (qtime, rtime, ttime) with host CPU in the same dashboard shortens the “is it the proxy or the backend?” step that opens every skew investigation.

Related guides

• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy connect time (ctime) high: network latency and backend accept-queue overflow
• HAProxy 502 Bad Gateway: backend connection and response failures
• HAProxy 503 Service Unavailable: no server is available to handle this request
• HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade
• HAProxy backend losing servers: active server count and cascade risk
• HAProxy connection errors (econ): backend connections refused, timed out, or reset
• HAProxy 5xx delta: telling HAProxy-generated errors from backend errors
• HAProxy health checks green but the application is broken: when UP does not mean healthy
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable
• HAProxy health check L7STS: server DOWN on the wrong HTTP status