HAProxy server flapping: rise, fall, and health checks oscillating UP and DOWN

A backend server that alternates between UP and DOWN is worse than one that is cleanly down. Every transition redistributes traffic: when the server goes DOWN its connections move to the survivors, when it comes back UP it absorbs a fresh share of load before it has warmed up, and every cycle costs retries, redispatches, and connection churn. If the flap rate is high enough, the oscillation itself becomes the incident.

The signature is easy to spot in the stats: chkdown keeps incrementing, lastchg (seconds since the last status change) never grows large, and the status field cycles through UP, DOWN 1/3, DOWN, UP 1/3 and back. Meanwhile chkfail climbs even while the server still shows UP, because individual checks are failing below the fall threshold.

This guide covers how to confirm flapping, how to read the health check counters to find the cause, how to stop the churn safely while you investigate, and how to tune rise, fall, inter, and the check timeout so a borderline server stops oscillating.

What this means

HAProxy’s health check engine runs independently of traffic. Each server is probed every inter milliseconds (default 2000). The state transitions are threshold-based:

  • A DOWN server must pass rise consecutive checks (default 2) before it returns to service.
  • An UP server must fail fall consecutive checks (default 3) before it is marked DOWN.

The transitional states appear in the stats as UP 1/3 or DOWN 2/3 style values: the fraction shows progress through the rise/fall counter. Flapping means the server lives on the boundary: it fails fall checks, goes DOWN, the load lifts off it (or the transient fault clears), it passes rise checks, comes back UP, receives traffic again, and fails again. Each cycle is a few inter periods long, so with defaults a full flap can complete in under 20 seconds.

flowchart TD
  UP["UP (serving traffic)"]
  TDN["DOWN n/fall (transitional)"]
  DOWN["DOWN (no traffic)"]
  TUP["UP n/rise (transitional)"]
  UP -->|"fall consecutive check failures"| TDN
  TDN -->|"threshold reached"| DOWN
  TDN -->|"a check passes"| UP
  DOWN -->|"checks pass"| TUP
  TUP -->|"rise consecutive successes"| UP
  TUP -->|"a check fails"| DOWN

Why this matters operationally:

  • Traffic churn. Every DOWN transition redistributes the server’s share of traffic to survivors. If the survivors are near capacity, the extra load pushes their response times up, which can make their own checks slower. This is how one flapping server starts a collapse cascade.
  • Retry noise. Requests in flight when the server drops produce wretr and wredis events and latency outliers, even when users never see an error.
  • Counter semantics hide the real failure rate. chkfail only increments for checks that fail while the server is UP. Once the server is DOWN, further failed checks do not move chkfail. A flapping server can therefore look less broken than it is if you only watch chkfail. chkdown is the counter that tells the truth about oscillation.

Common causes

CauseWhat it looks likeFirst thing to check
Check timeout near actual check latencycheck_duration close to the effective check timeout; last_chk alternating between OK and timeout codesCompare check_duration to timeout check (or inter if timeout check is not set)
rise/fall too tight for a jittery serverServer passes most checks but occasionally fails 2-3 in a row under loadCount check failures vs successes in logs; check whether failures cluster at traffic peaks
Server on the edge of overloadFlapping correlates with traffic peaks; rtime and scur/slim on the server rise before each DOWNPer-server rtime, scur, and application-level CPU/memory on the server
Intermittent packet loss or network instabilitylast_chk shows L4TOUT or L4CON without the server being loaded; other servers on the same path may flap tooctime trend on the backend; retransmits on the HAProxy host (ss -ti), network device counters
Application-level intermittent failurelast_chk shows L7STS or L7RSP (wrong status or wrong response body) while the process stays aliveThe health endpoint itself: does it depend on a downstream (database, cache) that is intermittently slow?
Health endpoint slower than real trafficChecks time out under load while the app still serves requestsWhat the health endpoint actually does; whether it shares a thread pool or connection pool with real requests
Passive checks (observe layer4/layer7) with aggressive on-errorFlapping with real traffic errors feeding back into health stateobserve, error-limit, and on-error settings on the server line

Quick checks

All of these are read-only against the runtime socket. Column numbers below match the standard show stat CSV field order; the first line of show stat output is a header row you can use to verify positions on your version.

# Status, check counters, and last change time for every server
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" && $2 != "BACKEND" {print $1"/"$2": status="$18" chkfail="$22" chkdown="$23" lastchg="$24"s last_chk="$57}'

# Detailed per-server state (operational vs configured state)
echo "show servers state" | socat unix-connect:/var/run/haproxy.sock stdio

# Check latency: is check_duration near the check timeout?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" && $2 != "BACKEND" {print $1"/"$2": check_duration="$39"ms"}'

# Is the flapping server also slow or saturated on real traffic?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" && $2 != "BACKEND" {print $1"/"$2": scur="$5" slim="$7" rtime="$61"ms"}'

# Retry and redispatch pressure caused by the flapping
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": wretr="$16" wredis="$17}'

Two things to look at immediately in the first command’s output:

  • lastchg resetting repeatedly. If lastchg never climbs past a few tens of seconds, the server keeps changing state. A stable server has a lastchg that grows for hours or days.
  • last_chk result codes. These tell you the failure mode directly: L4TOUT (TCP connect timeout), L4CON (connection refused), L6TOUT (TLS timeout), L7TOUT (response timeout), L7STS (wrong HTTP status), L7RSP (unexpected response content), L7OK (passing). The flap between an OK code and a specific failure code is your root-cause pointer. If you want just the bare code, check_status (column 37) carries it without the longer description.

How to diagnose it

  1. Confirm it is flapping, not just down. Take two samples of status, chkfail, chkdown, and lastchg a minute apart. chkdown incrementing between samples confirms UP to DOWN transitions are happening right now. chkfail incrementing while status stays UP means the server is failing individual checks below the fall threshold and is one bad stretch away from another transition.

  2. Read last_chk and classify the failure. Timeout codes (L4TOUT, L6TOUT, L7TOUT) point at latency: the check itself, the network path, or an overloaded server. L4CON points at the server refusing connections (process down or at a connection limit). L7STS/L7RSP point at the application answering but answering wrong.

  3. Compare check_duration to the effective check timeout. If timeout check is set, that is the read timeout for checks. If it is not set, inter doubles as the whole check timeout, so a server whose checks occasionally take longer than inter will produce spurious failures. A check_duration hovering near the timeout is the classic borderline case.

  4. Correlate flap timing with load. Look at the server’s rtime and scur/slim around the transitions. If the server flaps only during traffic peaks and its real-request latency climbs right before each DOWN, it is overloaded, and the health check is correctly detecting that. The fix is capacity or per-server maxconn, not check tuning.

  5. Rule out the network. If last_chk shows L4TOUT but the server’s own metrics look idle, check ctime on the backend (rising connect time suggests congestion), and look for retransmits on the HAProxy host with ss -ti. If several servers behind the same network path flap together, suspect the path, not the servers. Simultaneous multi-server flapping is almost always network or a shared dependency.

  6. Inspect the health endpoint itself. For HTTP checks, hit the exact check URL from the HAProxy host repeatedly and time it. If the endpoint touches the database or a cache, it can fail while the application still serves cached or simple requests fine. If it is a trivial static response, the opposite problem applies: the server can pass checks while real requests fail (see Health Check Green, App Broken).

  7. Stop the churn before it hurts the survivors. If the flapping is causing visible traffic churn (rising wredis, survivor scur climbing), put the server in maintenance mode while you investigate:

# Disruptive for that server: stops all new traffic to it. Existing sessions finish.
echo "set server <backend>/<server> state maint" | socat unix-connect:/var/run/haproxy.sock stdio

MAINT stops the redistribution oscillation completely. The survivors get a stable traffic share and you get a quiet system to debug on. Do not forget the server is in MAINT; a forgotten MAINT server is silent capacity loss. Bring it back with state ready when done.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
status (per server)The routing decision; transitional values (UP 1/3, DOWN 2/3) show rise/fall progressStatus changing at all on a normally stable server
chkdownCounts UP to DOWN transitions; the definitive flapping counterAny increment on a server that should be stable; increments on multiple servers at once (escalate)
chkfailIndividual failed checks while UP; leads transitionsClimbing while status still UP: the server is borderline
lastchgSeconds since last state change; a flapping server never lets it growRepeatedly resetting below a few hundred seconds
last_chkThe actual failure code; classifies the root causeOscillation between an OK code and a timeout/refusal/status code
check_durationLatency of the check itselfApproaching timeout check (or inter if unset)
wretr / wredisHAProxy masking the instability from usersRising in step with the flap cycle
Per-server scur/slim, rtimeTells you whether the server is actually overloaded when it failsSaturation or latency spike right before each DOWN
Survivor scur, backend qcurMeasures the blast radius of each redistributionSurvivors approaching their limits during flaps

Fixes

Stop the churn first (MAINT)

If the flapping server is actively destabilizing the backend, put it in MAINT via the runtime API as shown above. This is a traffic-affecting action for that server, so do it deliberately, and track how long it stays in MAINT. It converts an oscillating problem into a stable one you can reason about.

Check timeout too tight: set timeout check explicitly

If check_duration is close to the timeout, the common misconfiguration is a low inter set for fast detection with no separate timeout check. Because inter then serves as the entire check timeout, you have made the detection interval and the failure threshold the same number: any check that merely runs slow counts as failed. Set timeout check to a value comfortably above the p99 of check_duration but below inter, so a slow check fails but a normal one never does. Tradeoff: longer check timeouts slow down detection of genuine failures slightly.

Rise/fall too tight: widen the thresholds

Defaults (rise 2, fall 3, inter 2000) mean a server goes DOWN after roughly 6 seconds of consecutive failure and returns after roughly 4 seconds of success. For a server with jittery but recoverable behavior, that is a hair trigger. Raising fall to 5-10 (or spacing checks with a larger inter) makes a DOWN decision require sustained failure, and raising rise makes re-admission require sustained health. Tradeoff: you accept slower detection of real failures and traffic hitting a genuinely dead server for longer. Tune against your actual failure-detection budget, not against a desire to make the flapping counter quiet.

If many servers share the same check schedule and hardware, spread-checks in the global section adds jitter to check timing and avoids resonance effects where checks synchronize and pile up.

Server genuinely overloaded: fix capacity, not the check

If the server flaps because it is overloaded under its traffic share, health check tuning only masks it. Options: give the server a per-server maxconn so HAProxy queues instead of overwhelming it (see HAProxy maxconn hierarchy), add capacity, or fix whatever makes it slow (rtime decomposition, application profiling). In this case flapping was the health check working as designed.

Intermittent network: fix the path

L4TOUT/L4CON with idle servers and rising ctime means the path between HAProxy and the backend is dropping or delaying packets. Check retransmits on the HAProxy host (ss -ti), the interface error counters, and any middlebox (firewall connection tracking tables are a classic culprit for intermittent refusal). No amount of rise/fall tuning fixes a lossy path; it only changes how loudly HAProxy complains about it.

Passive checks amplifying the problem

If the server uses observe layer4 or observe layer7, real traffic errors feed into the health state via error-limit (default 10) and on-error. The default on-error fail-check simulates a failed check and forces the fast interval, which can accelerate a flap under a burst of real errors. If passive checks are causing oscillation, consider relaxing error-limit or removing observe until the underlying instability is fixed.

Prevention

  • Set timeout check explicitly on every checked server so check timeout is decoupled from inter. Never let a fast inter silently become a short timeout.
  • Choose fall and rise from your detection budget, and make rise strict enough that a server is demonstrably stable before it gets traffic back. A server that just recovered should prove it.
  • Baseline chkdown and lastchg per server and alert on any chkdown increment on servers that are normally stable for weeks. Flapping alone warrants an alert; it becomes urgent when survivors approach saturation.
  • Alert on check_duration approaching the check timeout. That is the earliest possible warning: the server is borderline before a single transition happens.
  • Watch the blast radius. During any flap, track survivor scur/slim and backend qcur. If one server’s DOWN transitions push survivors over 80%, the real fix is headroom, and you are one flap away from a cascade (see HAProxy backend queue building).
  • Make health endpoints check what real traffic needs, but keep them fast. A health endpoint that depends on a slow downstream will flap when the downstream does.

How Netdata helps

  • Per-server chkfail, chkdown, and lastchg as time series turns flapping from a log-grep exercise into a visible oscillation pattern: you see the flap frequency and whether it is accelerating.
  • check_duration trended against the configured timeout surfaces the borderline server days before the first DOWN transition, which is the cheapest time to fix it.
  • Correlating chkdown with wretr/wredis and survivor scur shows the cost of each flap: whether the redistribution is being absorbed cleanly or pushing the backend toward queuing.
  • lastchg never growing is a simple, robust alert condition for oscillation that does not depend on thresholds you have to tune per service.
  • Backend-level correlation (qcur, rtime, 5xx) against flap events distinguishes “server is broken” from “server is the first victim of a shared dependency problem”.