HAProxy queue time (qtime) rising: the earliest backend-saturation signal

You are looking at a backend whose qtime has crept from zero to a few hundred milliseconds, or a few seconds, and nothing is “down” yet. Health checks are green, error rates are flat, and no alert has fired. That is the point of qtime: it moves before the failure signals.

Nonzero qtime means requests spent time in HAProxy’s queue because every usable server connection slot was busy when they arrived. It is usually the first measurable symptom of backend saturation: ahead of 503s, ahead of timeout server expiries, ahead of user complaints.

What this means

HAProxy reports qtime on BACKEND and SERVER rows in the stats CSV. It is the average time, in milliseconds, that requests spent queued before dispatch. Two properties matter:

• It is a rolling average over the last 1024 requests, not a gauge. A short burst can be diluted by hundreds of non-queued requests. One very old queued request can also keep the average elevated until it leaves the window.
• Zero does not mean no queue ever formed. It means the recent average is zero. For queue depth right now, use qcur; for peak depth, use qmax.

The queue exists because HAProxy enforces connection limits. When the selected server, backend, frontend, or process has reached its maxconn, new requests wait for a slot instead of being rejected immediately. Queuing is backpressure; qtime is the latency clients pay before it becomes a hard failure.

qtime is one component of total session time:

flowchart LR
  A[Request arrives] --> B{Free server slot?}
  B -->|yes| C[connect: ctime]
  B -->|no| Q[wait in queue: qtime]
  Q --> C
  C --> D[server processing: rtime]
  D --> E[data transfer]
  E --> F[ttime]

The working identity is ttime ~= qtime + ctime + rtime + transfer time. When ttime rises but rtime is flat, the added latency is in queueing or connecting. qtime tells you which. In mode tcp, treat it as connection queuing, not HTTP request queuing.

Common causes

Quick checks

Read-only checks against the stats socket. Adjust the socket path. CSV column numbers are version-sensitive; if output looks shifted, compare fields against the show stat header line.

# Current qtime per backend and server (rolling avg, ms)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$1 !~ /^#/ && $2 != "FRONTEND" {print $1"/"$2": qtime="$59"ms"}'

# Real-time queue depth and historical peak
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$1 !~ /^#/ && $2 != "FRONTEND" {print $1"/"$2": qcur="$3" qmax="$4}'

# Session utilization at frontend, backend, and server rows
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$1 !~ /^#/ && $7>0 {print $1"/"$2": scur="$5" slim="$7" pct="int($5/$7*100)"%"}'

# Decompose backend latency: where is the time going?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$1 !~ /^#/ && $2 == "BACKEND" {print $1": qtime="$59" ctime="$60" rtime="$61" ttime="$62}'

# How many servers are actually serving?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$1 !~ /^#/ && $2 == "BACKEND" {print $1": active="$20" backup="$21}'

# Are retries or redispatches masking instability?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$1 !~ /^#/ && $2 != "FRONTEND" {print $1"/"$2": wretr="$16" wredis="$17}'

How to diagnose it

1. Confirm the queue is real and current. Check qcur with qtime. Because qtime is a rolling average, it can stay nonzero after the queue drains. qcur > 0 means requests are waiting now. qmax shows how deep it got.

2. Express qtime as a fraction of the time budget. If timeout server is 30s and qtime is 15s, half the budget is gone before the request reaches a server. A request that queues for most of timeout server is a timeout waiting to happen. If timeout queue is set, it caps queue wait before HAProxy returns a 503; compare qtime against that value too.

3. Find which limit enforces the queue. Check scur/slim at all four levels: global (CurrConns/Maxconn in show info), frontend, backend, and per-server. The limits are independent; a per-server limit can queue requests while global headroom looks fine. The level pinned near 100% is the bottleneck.

4. Separate slow servers from insufficient slots. If rtime rises with qtime, slots are held longer and effective pool capacity shrank. If rtime is flat and req_rate is up, demand exceeded dispatch capacity. The first is a backend-performance problem; the second is a capacity problem.

5. Check server count. If act dropped after health-check failures or maintenance, the same traffic is spread across fewer servers. Survivor overload shows as high per-server scur/slim on the remaining servers. This is the early stage of the backend-collapse pattern on the HAProxy monitoring checklist.

6. Check for masking signals. Sustained wretr/wredis means HAProxy is compensating for intermittent connect or dispatch failures, adding latency and holding slots. Watch frontend hrsp_5xx: once queued requests expire against timeout server or timeout queue, 503/504s appear. Rising qtime plus rising 5xx means you are past early warning.

7. Rule out TCP-mode artifacts. In TCP mode, qtime covers connection queuing. In TCP mode with content inspection, such as SNI-based routing, some deployments report qtime reflecting waits for initial client data rather than true slot starvation: nonzero qtime with qcur at zero. If qcur is consistently zero while qtime reads nonzero on a TCP frontend, suspect this before declaring saturation.

Metrics and signals to monitor

Fixes

Group fixes by the limit identified in step 3.

Per-server maxconn is the limit

Raise per-server maxconn only if the application can handle more concurrency. The limit protects the application; raising it moves the queue from HAProxy into the application’s worker pool or connection backlog, which is usually worse. Verify server-side headroom first: worker utilization, connection pool usage, GC or lock stalls, downstream latency. If the application is the constraint, fix the application or add servers. Tradeoff: higher per-server maxconn increases memory and FD use on HAProxy and the backend.

Too few servers or too little capacity

Add servers, or reduce per-request cost so slots turn over faster. If rtime is the driver, the fix belongs in the application or its dependencies: slow queries, GC pauses, downstream calls. Temporarily shed noncritical routes with ACLs to buy time. If servers are flapping, stabilize them to restore capacity; see HAProxy server flapping.

Backend or global maxconn is the limit

If the BACKEND row is pinned at slim while individual servers have headroom, the backend-level limit is too low for the aggregate pool. Review the maxconn hierarchy before changing anything, because limits interact across four levels. Raising global maxconn requires file-descriptor headroom (Maxsock, /proc/<pid>/limits) and memory headroom for connection buffers and metadata.

Bound the queue deliberately

When maxqueue is 0, the default, the queue can grow without an explicit depth cap and clients absorb delay until timeout server or timeout queue fires. Setting a bounded maxqueue and a sane timeout queue converts unbounded latency into fast, visible 503s, often the better failure mode for interactive traffic. Tradeoff: you trade graceful degradation for explicit errors, so alert on the 503 rate.

Do not restart HAProxy to clear a queue. The queue is a symptom of demand exceeding dispatch capacity. A restart resets counters, drops connections, and the queue rebuilds if the cause remains.

Prevention

• Alert on sustained nonzero qtime, not on errors. Express the threshold relative to timeout server, for example qtime above 10% of the server timeout for several minutes. This fires before users are affected.
• Alert on the qtime/qcur pair. qcur confirms the queue exists now; qtime tells you the cost. Either alone is easy to misread.
• Track qmax trends over weeks. Creeping peak queue depth is a capacity-planning signal: every traffic spike consumes more headroom.
• Capacity-test per-server maxconn against the real application limit. HAProxy’s limit should sit at or below what the application handles before degrading.
• Maintain N+1 server capacity per backend. With one server removed, survivors should still serve peak traffic without queuing.
• Handle counter resets. qtime and friends restart their windows on reload; monitoring must not read post-reload zeros as recovery.

How Netdata helps

• Netdata collects the HAProxy stats CSV continuously, so qtime, qcur, and qmax are per-second time series per backend and per server, not point-in-time snapshots from show stat.
• Because qtime is a rolling average, correlation matters: Netdata charts qtime next to qcur, scur/slim utilization, rtime, and hrsp_5xx so you can see the sequence “queue forms, latency rises, errors begin” in one view.
• Per-server breakdowns separate one saturated server from whole-pool exhaustion.
• ML-based anomaly detection on qtime and rtime can flag the low-amplitude early phase that fixed thresholds miss.
• Long retention on qmax and utilization trends supports N+1 capacity review without a separate metrics pipeline.

Related guides

• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy maxconn hierarchy: global, frontend, backend, and per-server limits
• HAProxy maxconn reached: new connections queued and rejected
• HAProxy health checks green but the application is broken: when UP does not mean healthy
• HAProxy server flapping: rise, fall, and health checks oscillating UP and DOWN
• HAProxy Slowloris and idle-connection pile-up: high scur, low request rate
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable
• HAProxy health check L7STS: server DOWN on the wrong HTTP status
• How HAProxy actually works in production: a mental model for operators
• HAProxy monitoring checklist: the signals every production proxy needs
• HAProxy monitoring maturity model: from survival to expert