HAProxy retries and redispatches (wretr/wredis): the backend instability nobody watches

Your dashboards are clean. No 5xx spike, latency averages look normal, every backend server is UP. Then the backend falls over in what feels like seconds, and the postmortem shows the application was flapping for six hours before the outage. The warning was there the whole time, sitting in two counters almost nobody charts: wretr and wredis.

These counters record HAProxy’s built-in compensation mechanism at work. When a connection to a backend server fails, HAProxy does not immediately return an error to the client. It retries. If option redispatch is set, it can also give up on the original server and dispatch the request to a different one. From the client’s perspective the request succeeds, possibly a bit slower. From the proxy’s perspective, something in your backend just failed and got papered over.

That papering-over is exactly why retries matter. High retries with zero user-visible errors means HAProxy is successfully masking backend instability. High retries with rising errors means the masking is failing and user impact has begun. The transition between those two states is what makes outages feel sudden: the failure was visible in the retry counters for hours before it reached the error counters. For the broader mental model of how requests flow through HAProxy, see how HAProxy works in production.

What this means

Two cumulative counters on BACKEND and SERVER rows in the stats CSV:

  • wretr (CSV index 15, awk field $16): the number of times HAProxy retried a connection to the same server after a failure.
  • wredis (CSV index 16, awk field $17): the number of times HAProxy redispatched a request to a different server than the one originally selected.

wredis only moves if option redispatch is enabled. Without it, HAProxy burns all of its retry budget on the same server and then errors out. The retries directive controls how many attempts HAProxy makes before giving up; the default is 3.

There are two operational states that matter:

StateWhat the counters showWhat it means
Compensation workingwretr/wredis rising, hrsp_5xx flat, latency slightly elevatedBackend is intermittently failing but HAProxy is hiding it. You have warning time.
Compensation failingwretr/wredis high, hrsp_5xx rising, rtime/qtime climbingRetry budget is exhausted often enough that errors leak through. Users are impacted.

Normally both counters are zero or near-zero. During a rolling deploy you will see brief, expected bumps as servers restart. Sustained retries above roughly 1% of total requests means significant backend instability, whether or not anyone has complained yet.

flowchart TD
  A[Backend intermittent failure] --> B[Connect or response fails]
  B --> C{Retry budget left?}
  C -->|yes, same server| D[wretr +1]
  C -->|redispatch enabled| E[wredis +1, try another server]
  D --> F{Recovered?}
  E --> F
  F -->|yes| G[Client sees success, extra latency]
  F -->|no, retries exhausted| H[5xx returned to client]
  G --> I[hrsp_5xx flat: compensation working]
  H --> J[hrsp_5xx rising: compensation failing]

Common causes

CauseWhat it looks likeFirst thing to check
Backend restarting or OOM-killing intermittentlywretr and econ rising together on one server, chkfail incrementing, lastchg smallBackend host logs, OOM killer in dmesg, container restart count
Rolling deployment or config reload on backendsShort, bounded bursts of wredis that self-resolve, servers cycling DOWN then UPDeploy pipeline timing vs counter increase
Server hitting per-server maxconnRetries correlate with scur at slim on one server, qcur nonzeroPer-server scur/slim in stats CSV
Network instability between HAProxy and backendsRetries spread across all servers in a backend, ctime elevated, econ on many serversctime trend, host network retransmit counters
Health check flapping near the fall thresholdchkfail rising but server stays UP, wretr tracks the failureschkfail rate and last_chk result code
Retry config masking, then amplifying, a slow backendwredis high, rtime and qtime climbing, latency tail growingWhether retries are hammering already-slow servers

Quick checks

All of these are read-only against the stats socket.

# Retry and redispatch counters per backend and server
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": wretr="$16" wredis="$17}'

# Correlate with connection errors and response errors
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": econ="$14" eresp="$15}'

# Which servers are flapping: status, check failures, time since last change
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" && $2 != "BACKEND" {print $1"/"$2": status="$18" chkfail="$22" chkdown="$23" lastchg="$24"s"}'

# Is the backend queueing as a side effect of slow, retried requests?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": qcur="$3" qtime="$59"ms rtime="$61"ms"}'

# Confirm whether redispatch is even possible in the running config
grep -E "redispatch|retries|retry-on" /etc/haproxy/haproxy.cfg

Two things to note when reading these numbers. First, wretr and wredis are cumulative and reset to zero on every reload, so compute deltas or rates before comparing anything. Second, the retry-to-request ratio is what matters: divide the retry delta by the request delta over the same window. Absolute counts are meaningless across backends with different traffic volumes.

How to diagnose it

  1. Establish the rate, not the count. Sample wretr and wredis twice, 60 seconds apart, and compute the delta per server. Compare against the request volume delta for the same backend. Anything sustained above ~1% of requests is significant instability.

  2. Identify the blast radius. Are retries concentrated on one server, or spread across the whole backend? One server points at that host (restarts, OOM, saturation). The whole backend points at something shared: network path, a common dependency causing timeouts, or per-server maxconn everywhere.

  3. Pair with econ and eresp. wretr rising with econ means HAProxy cannot connect: refusals, timeouts, resets. wretr rising with eresp means it connects but the response breaks. These are different failure modes with different fixes. The distinction is covered in the signal correlation rules of the HAProxy monitoring checklist.

  4. Check the flapping evidence. chkfail incrementing on a server that still shows UP means health checks are failing below the fall threshold. Real traffic is hitting a server that is already failing probes. last_chk tells you how the check is failing (connection refused vs timeout vs wrong status).

  5. Check whether compensation is holding. Compare frontend hrsp_5xx against backend hrsp_5xx. If the frontend error rate is still flat while retries climb, HAProxy is absorbing the problem and you have time. If frontend 5xx is climbing with the retry rate, the retry budget is being exhausted and the incident has already started for users.

  6. Check for retry-amplified latency. Retries add delay. Watch qtime and rtime: if they are climbing alongside retries, the compensation mechanism itself is degrading the user experience even while it hides errors. Retried requests against an already-slow backend also hold connection slots longer, which feeds the queue. See HAProxy backend queue building (qcur) for that failure mode.

  7. Verify the retry configuration matches your assumptions. Confirm option redispatch is actually set if you expect wredis to move, and check the configured retries count. A backend where you assumed redispatch was active but is not will show wretr climbing and errors arriving earlier than expected.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
wretr rate per serverSame-server retry volume; the raw instability counterSustained nonzero, or > 1% of that server’s requests
wredis rate per backendCross-server redispatch; shows failover is happening under loadRising trend outside deployment windows
econ rateConnect failures that trigger retriesRising alongside wretr
eresp rateBroken responses, a different root cause classRising alongside wretr with flat econ
chkfail / chkdownHealth check flapping that predicts traffic-side failureschkfail climbing while status stays UP
rtime / qtimeLatency cost of the compensation itselfClimbing in step with retry rate
Frontend minus backend 5xx deltaHAProxy-generated errors when retries exhaustDelta rising from zero
lastchg per serverRecent status transitionsSmall values correlating with retry bursts

Fixes

There is no “fix the retries” knob. The counters are the messenger. Fixes target the cause.

Backend restarting or crashing

Fix the restart loop: memory limits, application crash cause, deploy health. This is the most common root cause of a sustained wretr trend on a single server. Do not tune HAProxy retry settings to hide it; you are trading warning time for a harder failure later.

Per-server maxconn too low for the workload

If retries cluster on servers sitting at scur = slim, HAProxy is refusing new connections to full servers and retrying. Raise per-server maxconn to what the application can actually sustain, or add capacity. Raising maxconn without understanding backend capacity trades backpressure for overload. See the maxconn hierarchy guide.

Retry configuration not doing what you think

Several configuration behaviors routinely surprise operators:

  • Without option redispatch, there is no failover to another server. HAProxy retries the same server up to the retries count, then errors. If you expect wredis to be nonzero and it is pinned at zero during failures, this is why.
  • Redispatch timing is tunable. The optional interval argument to option redispatch controls when the redispatch happens within the retry sequence; the default behavior redispatches late in the retry sequence rather than on every retry. If you want a fresh server on the first retry, that has to be configured explicitly.
  • retry-on replaces the default, it does not extend it. On HAProxy 2.0 and later, setting retry-on 503 504 without including a connection-failure condition such as conn-failure means connection failures stop being retried. If wretr mysteriously drops to zero after a config change while econ keeps rising, check for this.
  • Non-idempotent requests are not retried by default. Layer-7 retries do not fire for methods like POST unless you explicitly allow it, because replaying a POST can duplicate side effects. That is the safe default; think carefully before changing it.

Network instability to backends

If retries spread evenly across servers with elevated ctime, the path is the problem: packet loss, congestion, an overloaded middlebox. Fix the path, not the proxy.

Rolling deployments

Do not “fix” expected retry bursts during deploys; suppress alert noise instead. Deploy windows should be annotated so a retry spike during a rollout is evaluated differently from the same spike at 3 a.m.

Prevention

  • Chart the counters, not just the errors. Add wretr and wredis rates per backend to the standard HAProxy dashboard. This is a Level 3 maturity signal for a reason: it converts a sudden outage into a multi-hour warning. See the monitoring maturity model.
  • Alert on the ratio, with compensation state. Alert when retry rate exceeds ~1% of requests sustained. Escalate severity when the retry rate is high AND frontend 5xx is rising: that is compensation failing.
  • Baseline around deploys. Know your normal retry envelope during rollouts so deviations stand out.
  • Test the config assumption once. Trigger a controlled backend failure and confirm wredis actually increments. Many teams discover mid-incident that option redispatch was never set.
  • Correlate with capacity signals. Retries that track per-server scur/slim saturation are a capacity problem wearing an instability costume. See scur approaching slim and backend losing servers.
  • Handle counter resets in your monitoring. Every reload zeroes wretr and wredis. Rate calculations must detect resets or you will see phantom drops and phantom spikes. Uptime_sec from show info is the standard reset detector.

How Netdata helps

Netdata’s HAProxy collector pulls the stats CSV continuously, so the retry story is visible without manual socket queries:

  • Per-server and per-backend wretr/wredis rates rendered as deltas, so cumulative counter resets on reload do not corrupt the trend.
  • Side-by-side correlation with econ, eresp, and per-server status, which is how you split “cannot connect” from “response broke” without leaving the dashboard.
  • Retry rate next to hrsp_5xx on the same backend, making the compensation-working vs compensation-failing distinction a visual one.
  • Retry rate overlaid with qtime/rtime, exposing the latency tax that compensation charges even when it hides every error.
  • Per-second granularity that catches short flapping bursts that minute-resolution scrapers average away.