HAProxy response time (rtime) climbing: slow backends behind the proxy

Your HAProxy dashboards show rtime creeping up on one or more backends. Clients have not started timing out yet, or maybe they just have. HAProxy itself looks fine: CPU is reasonable, no servers are DOWN, error rates are flat or only slightly elevated. But the average time between HAProxy forwarding a request and the backend’s first response byte is growing, and it is not coming back down.

rtime measures server processing time as the proxy sees it: request sent, first response byte received. When it climbs, the application behind the proxy is slowing down. The usual suspects are database queries, downstream dependency latency, resource contention, GC pauses, or thread pool exhaustion on the backend.

The trap is that rtime is a rolling average over the last 1024 requests, and it has a dangerous blind spot: a backend returning instant HTTP 500s shows low rtime. Fast errors look healthy in this metric. This guide walks through confirming the slowdown is real, isolating which server is slow, and ruling out the cases where rtime is lying to you.

What this means

HAProxy exposes four timing averages per backend and per server: qtime (queue wait), ctime (TCP connect to server), rtime (server processing), and ttime (total session time). The decomposition is the diagnostic tool:

ttime ≈ qtime + ctime + rtime + data transfer time

When rtime climbs while qtime and ctime stay flat, the time is being spent inside the backend application. HAProxy is dispatching requests promptly and connecting quickly; the server is just taking longer to answer.

Two properties of rtime matter before you act on it:

1. It is a rolling average over 1024 requests. Brief spikes get smoothed away, and a slowly rising average means the slowdown has been building for a while. For percentile-level visibility (P95, P99), you need per-request timing from HAProxy logs, not the stats counters.
2. It times responses, not correctness. A server failing fast with instant 500s produces excellent rtime numbers. A rising rtime with rising 5xx is a genuinely slow backend. A falling rtime with rising 5xx is a backend failing fast, which is a different incident. Always correlate with hrsp_5xx.

In TCP mode, rtime is 0 because there is no HTTP response to time. If you are looking at a TCP-mode backend, use ttime and connection-level signals instead.

Common causes

Quick checks

These are all read-only against the runtime socket. Adjust the socket path if yours differs.

# Per-server and per-backend rtime: find the slow node
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$1 !~ /^#/ && $2 != "FRONTEND" {print $1"/"$2": rtime="$61"ms"}'

# Decompose latency: queue, connect, response, total
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$1 !~ /^#/ && $2 == "BACKEND" {print $1": qtime="$59"ms ctime="$60"ms rtime="$61"ms ttime="$62"ms"}'

# Queue depth: is the slow backend starting to hold requests hostage?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$1 !~ /^#/ && $2 != "FRONTEND" {print $1"/"$2": qcur="$3" qmax="$4}'

# 5xx per server: rising 5xx with rising rtime = genuinely slow;
# rising 5xx with LOW rtime = failing fast
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$1 !~ /^#/ {print $1"/"$2": 5xx="$44}'

# Connection errors and retries: is the network actually fine?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$1 !~ /^#/ && $2 != "FRONTEND" {print $1"/"$2": econ="$14" wretr="$16" wredis="$17}'

# Client aborts: are users already giving up?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$1 !~ /^#/ && $2 != "FRONTEND" {print $1"/"$2": cli_abrt="$50" srv_abrt="$51}'

CSV column indices above follow the documented show stat field order (qcur=3, qmax=4, econ=14, wretr=16, wredis=17, hrsp_5xx=44, cli_abrt=50, srv_abrt=51, qtime=59, ctime=60, rtime=61, ttime=62). The header line (# pxname,svname,...) in your own show stat output is the ground truth; count against it before scripting around these fields, especially on older HAProxy versions.

How to diagnose it

flowchart TD
  A[rtime rising] --> B{All servers or one?}
  B -->|One server high| C[Inspect that node: host CPU, memory, disk, app logs]
  B -->|All servers high| D{Shared dependency?}
  D -->|DB / cache / API slow| E[Fix or shed load on the dependency]
  D -->|No obvious dependency| F[App-level contention: GC, thread pool, I/O]
  A --> G{hrsp_5xx rising too?}
  G -->|rtime LOW + 5xx high| H[Fast-failure: backend broken, not slow]
  G -->|rtime high + qcur rising| I[Timeout cascade building: act now]

Work through it in this order:

1. Confirm the shape of the slowdown. Pull qtime, ctime, rtime, ttime for the backend. If the increase is almost entirely in rtime, the backend application is the problem. If qtime is also climbing, slow servers are holding connection slots and the queue is building: you are further along the failure curve than the average alone suggests. If ctime is the component that moved, this is a network or server-accept problem, not application processing; see the connect-time path instead.

2. Compare per-server rtime. One node at 3-10x the others points at a host-level problem on that node: a bad disk, a noisy neighbor, a stuck GC, a half-failed deploy. All servers rising together points at something shared: the database, the cache tier, an external API, or a code deploy that made everything slower.

3. Correlate with 5xx before trusting the number. Compare frontend and backend 5xx rates. If backend 5xx is rising alongside rtime, the application is slow and erroring: classic overload or dependency failure. If backend 5xx is rising while rtime stays low or falls, the application is failing fast, and this article’s symptom is actually the “health check green, app broken” pattern: see HAProxy health checks green but the application is broken.

4. Check whether the cascade has started. The backend timeout cascade signature is: servers still UP, econ low, rtime climbing, qcur climbing, frontend 504s beginning. If you see qcur > 0 and rising alongside the rtime increase, treat it as an active incident, not a trend. Each slow response holds a server connection slot longer, which reduces effective capacity, which makes queueing worse. See HAProxy backend queue building (qcur).

5. Check the masking signals. Rising wretr/wredis with flat user-facing errors means HAProxy is compensating for intermittent backend instability and you are seeing the early stage. Rising cli_abrt means clients have already noticed and are disconnecting before responses arrive; that is a leading indicator of user impact and often moves before error rates do.

6. Drop to the logs for percentiles. The 1024-request average hides tail latency. HAProxy log lines carry per-request timing fields (Tq/Tw/Tc/Tr/Tt), which let you compute P95/P99 and see whether all requests got slower or a subset (one endpoint, one tenant) is dragging the average.

7. Move to the backend hosts. At this point HAProxy has told you everything it can: which backend, which server, how slow, whether the network is fine. The remaining investigation is on the application side: slow query logs, GC logs, thread pool metrics, disk latency, downstream call latency.

Metrics and signals to monitor

Fixes

One slow server

Drain it and investigate off the critical path. Use the runtime API rather than letting health checks flap the server in and out:

# Disruptive: stops new sessions to this server. Existing sessions finish.
echo "set server <backend>/<server> state maint" | socat unix-connect:/var/run/haproxy.sock stdio

# Softer alternative: drain mode, only for cases where you want to keep
# persistent connections alive while shifting new traffic away.
echo "set server <backend>/<server> state drain" | socat unix-connect:/var/run/haproxy.sock stdio

If the node recovers after investigation, bring it back (state ready) and watch its rtime against the pool for at least one full traffic cycle. If per-server rtime diverges again, the problem is on that host, not the application version.

Tradeoff: removing a node concentrates traffic on survivors. Check survivor scur/slim headroom first, or you trade a slow-backend problem for a cascade problem.

Shared dependency slow

This is the most common root cause when all servers degrade together. Options, roughly in order of safety:

• Shed load: rate-limit or defer non-essential traffic (batch endpoints, exports, reports) until the dependency recovers.
• Reduce per-server maxconn temporarily to create backpressure and stop connection pile-up on the backends. This converts silent degradation into visible queueing you can manage, and protects the backend from being pushed past the point where it can recover.
• Fail over or restart the dependency if that is within your operational authority and runbook.

Tradeoff: backpressure moves pain from “everything is slow” to “some requests wait or fail fast.” That is usually the right trade during a cascade, but it is a deliberate choice, not a free fix.

Application-level degradation (deploy, GC, thread pool)

If rtime jumped at a deploy boundary, roll back. Rollback is the fastest fix and the one you will wish you had done first. If the degradation is gradual with no deploy, engage the application team with the per-server rtime data and the log-derived percentiles you collected; “HAProxy says server-side time tripled on all nodes starting at 14:20” is a far better handoff than “the site feels slow.”

Timeout posture while you fix it

timeout server is the safety net: when rtime approaches it, HAProxy terminates the wait and returns 504s. Do not raise it to mask the problem. Raising timeout server during a slow-backend incident lets connections pile up longer, worsens queueing, and delays the 504s that at least free slots. The right direction during an active incident is usually the opposite: tighten admission (per-server maxconn, rate limits) so the backend can recover.

Prevention

• Alert on rtime as a ratio, not an absolute. Two thresholds work well in practice: rtime > 2x its rolling baseline (catch degradation early) and rtime > 50% of timeout server (margin is thin, escalate). Absolute thresholds are meaningless across services with different normal response times.
• Always pair rtime with hrsp_5xx in the same alert evaluation. A single “backend slow” alert that does not check error rates will both miss fast-failure incidents and misreport their direction.
• Watch per-server rtime, not just the backend average. Backend averages hide a single degrading node until it drags the pool.
• Monitor the masking signals: wretr/wredis and cli_abrt move before user-facing errors do.
• Set per-server maxconn deliberately so a slow backend creates visible backpressure (qcur, qtime) instead of silently absorbing unlimited concurrency and collapsing. The maxconn hierarchy guide covers how the four levels interact.
• Compute percentiles from logs on a standing basis. Averages over 1024 requests are a triage tool, not a latency SLO instrument.

How Netdata helps

• Netdata collects the HAProxy timing family (qtime, ctime, rtime, ttime) per backend and per server at per-second granularity, so you see the divergence between one slow node and the pool as it starts, not after the 1024-request average catches up.
• Correlating rtime with hrsp_5xx on the same dashboard makes the fast-failure trap visible immediately: latency dropping while errors rise is a distinct, obvious pattern.
• Queue depth (qcur) and queue time alongside rtime show whether a slow backend has progressed into the timeout-cascade stage, which changes the urgency and the fix.
• Retry and redispatch counters (wretr/wredis) plus cli_abrt surface the compensation layer: backend instability HAProxy is hiding from users, and the moment users start noticing.
• ML-based anomaly detection on rtime catches the slow drift case, where the average creeps 20-30% over days and never crosses a static threshold until the backend is in real trouble.

Related guides

• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy health checks green but the application is broken: when UP does not mean healthy
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable
• HAProxy health check L7STS: server DOWN on the wrong HTTP status
• How HAProxy actually works in production: a mental model for operators
• HAProxy maxconn hierarchy: global, frontend, backend, and per-server limits
• HAProxy maxconn reached: new connections queued and rejected
• HAProxy monitoring checklist: the signals every production proxy needs
• HAProxy monitoring maturity model: from survival to expert
• HAProxy server flapping: rise, fall, and health checks oscillating UP and DOWN
• HAProxy Slowloris and idle-connection pile-up: high scur, low request rate